Transcript why we - Untangle Fight Club

Antivirus Fight Club!
August 8th @ LinuxWorld
Before we begin…
Please submit any viruses for the test at http://virus.untangle.com/
Background - who we are
Untangle provides an open source network gateway platform.
We are not an antivirus company
We are not a testing company
Background - why we are doing this
• 2005: Untangle researches antivirus to add to the network gateway platform
• after testing we choose clam (open source) and one other vendor
• 2006: Untangle seeks Testing Labs for certification (stickers!)
• 2006: Testing Lab refused to test AV product, because use of open source
• won’t tell us why
• won’t provide test results
• won’t provide test set
Something fishy is going on here…
What is the AV FightClub?
A simple test of real-world anti-virus detection by different AV engines
What AV FightClub is not:
• Zero-day test
• Functionality comparison
• Not coverage testing
Two important things!
• Open - for samples & participation & discussion
• Transparent - simple, verify & run at home
The Test
Each vendor is subjected to:
• Small Set of test viruses (eicar)
• Set of ‘in-the-wild’ viruses
• Set of user-submitted viruses (minus non-viruses, not ‘in-the-wild’ viruses, and phish)
Scored by % of viruses identified and performance if applicable
All vendors should catch all these viruses
The Vendors
Vendors
Engines with linux support (clam, kasperskey, fprot, sophos, globalhauri)
Gateway Appliances (sonicwall, fortinet, watchguard)
Windows solutions (norton/symance, mcafee)
Questions?
predictions?
Lets get started
• zip up the test set for windows tests
• deposit on web server for gateway appliance tests
F-Prot
Vendor
Version
4.6.8
Updated
2007-08-08
Method
Linux Client
Sophos
Vendor
Version
4.20.0
Updated
2007-08-08
Method
Linux Client
GlobalHauri
Vendor
Version
SDK 4.0. engine 2007-08-07
Updated
2007-08-08
Method
Linux Client
Kasperksy
Vendor
Version
Kav4fs 5.5.27
Updated
2007-08-08
Method
Linux Client
Norton/Symantec
Vendor
Version
Norton Antivirus 2007
Updated
2007-08-08
Method
Windows Client
McAfee
Vendor
Version
7.2.147
Updated
2007-08-08
Method
Windows Client
Sonicwall
Vendor
Version
Sonicwall 1260 (3.2.0.5-54e)
Updated
2007-08-08
Method
Gateway Appliance
Fortinet
Vendor
Version
Fortinet 50A (2.8.0-520)
Updated
2007-08-08
Method
Gateway Appliance
Watchguard
Vendor
Version
Watchguard x20e (8.5.1-8138)
Updated
2007-08-08
Method
Gateway Appliance
Clam
Vendor
Version
0.91-1-1ubuntu3
Updated
2007-08-08
Method
Linux Client
Results
Wild+Eicar Catch Rate
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Catch Rate
Results 2
Overall Catch Rate
100.0%
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
Catch Rate
Conclusions
conclusions
• Open Source solution (Clam) doesn’t suck . In fact, its excellent!
• Many vendors are poor. Some are selling dead donkeys!
outstanding questions
• Why hasn’t this been pointed out?
• Is there something wrong with the way we test antivirus today?
Thanks for coming!
Contact
Dirk Morris
[email protected]
Remember
Don’t believe me? Try this at home.
The test set will be available on http://virus.untangle.com
(password on zip file is “a”)