Transcript Cerberian’s Overview of Process to Capture & Rate Websites

Mail-Filters Technical
Presentation
How it works, Why it’s Better
Mail-Filter Technology Overview
•
•
•
•
•
•
•
•
•
•
Why Mail-Filters
Bullet Signature Creation
Star Engine Process Overview
Implementation Options
SDK Contents
Getting Started
The API Commands
Testing Options
OEM Implementation Examples
FAQs
Why Mail-Filters
• It’s Fast – 100s of messages per second (or higher)
• It’s Accurate – over 95% of spam caught, less than 1 in
1,000,000 false positive rate
• Many implementation options – the right solution for any
environment
• It’s Proprietary – it’s not fooled by spammer tricks - gives
time to market and competitive differentiation
• It catches Foreign Language Spam – in over 30
languages – a worldwide solution
• Easy Implementation – usually less than a day
• Full Support – Integration, technical support and training,
marketing materials, sales training and lead generation
How Mail-Filters Works
1. Spam Collection occurs
from many sources
5. Tuning Users and Administrators
provide feedback to help identify
spam and those that send them.
2. Human Editors Craft Bullet
Signatures
3. Bullet Signatures Are
Updated Every 1-15 Minutes
4. Mail-Filters Technology Integrated into
OEM Solutions - Catches Spam, without False
Positives
Mail-Filters’ Process Overview To Capture
Spam & Create Bullet Signatures
Mail-Filters Data Centers
Bullet Signature Updater
Customer
Customer submissions
Bullet
Signatures
Traffic and Connection
Heuristics
Mail-Filters
Technology on
Customer Device
Auto-Nominate
Process
Phish Trolling
Spam
DB
International Spam
Harvester
Culling
Engine
Bullet
Signature
Updates
Scam Sensors
www
www
Phish Traps
Traffic Analysis
Language
Assignment
Partner Collections
Prioritization
Process
Aristotle
(Signature
Auto-Suggest)
Spam Pre-Qualification
Partner Pre-Qualification
Expert
Pre-Qualified &
Auto-Nominated
Reputation
Analysis
Data Quality
Manager
Bullet Signature Creation
Spammer Profile Creation
Translation Tools
Human Editors
Message Profile Creation
Traffic Profiles
Quality
Check
Star Engine Process Overview
Mail-Filters
Data Centers
STAR Engine Server
OEM
Software
Is Message Spam?
Star Engine Interface
Yes / No
STAR Engine Management Module
Known Good Mail
Message
Normalizer
SnowFlake Buster
Language Analyzer
Bullet Signature
Updater
Malformed Message
Processor
Message Analysis
Traffic Analysis
Reputation Analysis
Spammer Profile Check
False Positive
Rationalizer
Bullet
Signatures
Implementation Options
• Enterprise
– Most typical implementation – highest
performance – uses more resources
• Desktop
– Small footprint – message is local – scan and
database is remote
• Embedded
– Tiny amount of resources required – scanning
is done remotely
Star Engine – Enterprise
(Very High Performance)
API
OEM Application
C or C++
• The SEI and SES are typically
deployed on the same hardware
• The SEI is linked into the OEM
application using C or C++
• The SES runs as a Service or
Daemon and it manages it’s own
Database Updates
• The Database is usually between
3-10MB – will download a fresh
DB upon startup if none present
Linked Together by OEM at compile
Star Engine Interface (SEI)
TCP / IP
changes are downloaded)
Server or Appliance
Hardware
Star Engine Server (SES)
(Service or Daemon)
TCP / IP
• Can process 100s or even over
1000 messages per second
• Requests Bullet Signature
updates every 1-10 minutes (only
Mail-Filters
Data Centers
Star Engine - Enterprise
• The Star Engine Server is fully multi-threaded
• The Star Engine Server will run as a Service
under Windows or as a Daemon under Linux,
FreeBSD, or Solaris
• TCP/IP outbound on Port 80 is required – IP
proxies are supported
• Typical requirements are P4, 100MB RAM, Hard
Disk optional
• A unique Mail-Filters Customer ID is required to
download the Bullet Signature Database
Star Engine – Desktop
(Small Footprint)
PC or Other Device
(with limited resources)
Linked Together by OEM at compile
API
C or C++
OEM Application
TCP / IP
Star Engine Interface
Separate Server
Star Engine Server
TCP / IP
• Only requires 128kb of RAM
• Can process 10s of messages
per second
• Secondary server can be
anywhere, including and
typically Mail-Filters’ Data
Centers
• Database updates are not
required on the SEI (just the
SES)
• Same exact API as the
Enterprise implementation
• Can also be used in a server
cluster environment – many
SEI’s feeding one SES
Mail-Filters
Data Centers
Star Engine – Embedded
A Completely New Approach
•
•
•
•
•
•
•
Anti-Spam detection for edge devices with
almost no resource requirements
OEM code requires less than 10kb of
RAM
No software need be installed on any user
PC – the service is turned on or off at the
OEM device
Works with POP3 & IMAP
OEM device intercepts the message
delivery request and sends it to MailFilters
Mail-Filters receives the messages on
behalf of the end user, filters for viruses
and spam, then sends the clean
messages to the end user
OEM or customer determines what
happens to spam (delete, mark with an X-
Email Server
4. Mail-Filters’ authenticates as the
user to the ISP or Corporate email
servers - the mail is delivered
3, Mail-Filters makes the request on
behalf of the user, filters the
messages, then sends the good mail
to the user. No mail is kept at MailFilters – it just passes through.
WWW
Mail-Filters
Data Centers
2. OEM device intercepts the request
based on port the request is made on
(Ex. 110 = POP3) – and redirects the
request to Mail-Filters’ data centers.
header, decorate the subject line)
•
Since spam can be deleted and the
downlink speed is probably slower than
the link from Mail-Filters’ data centers to
the email servers – good mail will get to
the end user faster.
1. Email Client requests
mail
PC
Embedded Architecture
Email Server
Mail-Filters
Data Centers
The Internet
OEM Device
Redirect Code
OEM Application
Outbound Listening Code (Port 110 for POP3 or Port 147 for IMAP Requests)
Customer
Premise
Email Server
PCs
The Email Client requests email from an email server – it makes the request on
port 110 or 147 – the OEM device redirects the request to Mail-Filters. A port is
opened by the email server via Mail-Filters to the PC. The email is filtered, a
policy is applied, then delivered to the Email Client.
SDK Contents
• Star Engine Server software executables
• Star Engine Interface libraries in C and
C++
• Simple Single-Threaded implementation
example application
• Documentation
• Typical integration time is less than a day
Getting Started with the SDK
• Install the Star Engine Server
• Run the Star Engine Server
• Run the Example Application
– This application will scan the files in the
directory of choice and all sub-directories to
see if they are spam. The results will display
on the screen.
• Begin the Integration to the OEM
application
The Star Engine API
(The Star Engine Interface)
• The Commands are Straight-Forward
– Initialize – This command establishes a connection to
the Star Engine Server
– Shutdown – Used to tear down the thread after a
successful Initialize command
– Scan SMTP Buffer – Passes the SES the data to be
scanned – will return TRUE if Spam
– SCAN Buffer – Passes the SES data to be scanned –
best used for non-SMTP types of content such as IM,
SMS, web pages, etc.
– Version – Returns the versions of all the components
currently being used, including the database version
date.
Testing Options
• The Mail-Filters database is culled to eliminate old/unused signatures.
– As a result, the catch rate will suffer on old corpuses of email
– Best results are obtained with live (or very close to it) email.
• There are several options to test the Mail-Filters technology
– To test for catch rate or false positive rate
• Use the Example scan utility to check individual messages in a directory
• Send mail to an account Mail-Filters can set up for you at Cleantree.com.
Good mail will go to the Inbox, spam to the Spam folder. Check results using
your browser.
• Integrate into the OEM application and run it to check catch rate.
– To test throughput:
• Unfortunately, the Example application is only a single-threaded application
and will not show what the SES can achieve throughput-wise (it does fine on
catch rate)
• The only fair test is to do an integration and run email through it. Most OEMs
fine the solution throughput is the same whether Mail-Filters technology is
running or not.
– To test Foreign Language:
• Do a beta test with a customer or partner in the region of interest
• Mail-Filters have several partners in various regions that may assist in a beta
test, if desired.
Implementation Examples
• Enterprise
– Most OEMs have implemented the Mail-Filters
technology as the primary anti-spam solution
• AV solutions company scans for spam while it has the message
in memory to scan for viruses. Because spam is more prevalent
and is a much faster scan, spam is typically scanned for first.
– Some have augmented their own anti-spam technology
• Because Mail-Filters technology is both fast and accurate, some
have used it as a pre-processor to their own, more
computationally expensive technology, to increase the
throughput of the overall solution, and to increase spam catch
rates.
Implementation Examples
• Desktop
– Some devices don’t have the processing power or
resources available for spam detection. For these,
the Mail-Filters technology can provide a smaller
footprint
• Firewalls, security gateways, messaging gateways,
enterprise PCs may prefer a secondary server to handle the
scanning to free up resources on their own hardware.
– An MSP has a cluster environment where there are
many SEIs feeding one SES per tower. This is very
efficient and allows their overall throughput to
increase dramatically.
Implementation Examples
• Embedded
– Ideal for DSL routers, Cable Modems, Wireless
gateways, SMB security gateways etc.
– Because it requires no end user software
installation or configuration, it is simple to signup and have spam and viruses eliminated.
Frequently Asked Questions
• How do I get the SDK?
– Sign the Mail-Filters MNDA and we’ll send it to you via email.
• Is the Star Engine Server multi-threaded?
– Yes.
• Does it handle messages in double-byte character sets?
– Yes, our technology catches spam in over 30 languages,
including multi-byte character sets such as Japanese, Korean,
Chinese, Arabic, and Hebrew.
• How is the update interval set – can it be changed?
– The update interval is set by the OEM, but can be changed on a
customer by customer basis. The default is an incremental
every 10 minutes and a full update written to disk once a week.
• Will this solution work on less than a Pentium IV PC?
– Yes, but it works more efficiently on a PIV.
Frequently Asked Questions
•
What happens if the SES can’t get a database, or quits running, or some
other catastrophe?
– The SES or SEI will fail safe. It will return a FALSE ( the message isn’t spam)
and continue to process messages while trying to reconnect. The customer will
see more missed spam, but won’t miss any messages.
•
What if the SES doesn’t have the rights to write the database to disk, or the
disk is full?
–
•
The SES will continue to function properly and will acquire updates to the
database in memory. The version command will return the database currently
being used in RAM.
Is the API really just 5 functions?
– Yes – it doesn’t get much simpler than that.
•
Can the SES return a probability of a message being spam?
– No - Because the technology uses human editors to craft profiles and message
signatures, we’re very very confident the message is spam if we identify it.
Because our false positive rate is so low, our methodology is proven to be
correct. A probability is required by technologies that guess or compute whether
a message is spam – we know it, so we tell you. For those solutions that require
a probability, they set our TRUE response to the highest probability – 10 or 1 or
100.
Conclusions
• The Mail-Filters technology is easy to
implement and provides options for any
situation.
• The underlying technology far surpasses
what others are doing, giving the Mail-Filters
OEM a significant advantage over
competitors in catch rate and accuracy,
language coverage, and throughput.
• Human review provides the difference -the
technology delivers it.