Antispam From The Inside Out

Vadim Markovtsev, Mail department / Antispam team26.03.2016 @ MIPT

View this on your device

goo.gl/5x4FzW

Mail.Ru Mail in numbers

80 000 000 active users monthly (30 000 000 daily)
800 000 signups every day
500 000 000 emails (72 TB) every day
700 antispam servers
1 PB cluster for analytics, 2 PB logs storage
Largest mail service in Russia; international version is my.com

Mail.Ru Antispam from technical point

OS: CentOS (production), MacOSX (analytics)
Database: Tarantool, MySQL, memcached
Analytics: Hadoop, HBase, Pig, Hue
Monitoring: Graphite, Zabbix, Sentry
Core: C++ engine + Lua business logic
Services: Python, Lua, Java, Perl

Analytical tasks

Filtering
- Spam removal
- Bots detection
Security
- Password anti-bruteforce
- Break-in detection
Product

Operational tasks

Attack vectors
- Reaction
- Prediction
Performance improvement
- Profiling
- Debugging
Durability
- Fail-safety arrangements
- Refactoring

Spam tradeoffs

how much spam is allowed vs. how comfortable user is. system load vs. how well emails are analyzed.

Technologies

Machine Learning

A subfield of computer science that evolved from the study of pattern recognition and computational learning theory in artificial intelligence.

Wikipedia

In other words, real-life problems are stated as optimization problems (models) which are subsequently solved by various methods (machine learning algorithms) on computers without direct human participation.

Example: clustering

Big Data?

A term for data sets that are so large or complex that traditional data processing applications are inadequate.

Wikipedia

This may include:

Large datasets which do not fit into 1 computer
Large data streams which cannot be stored on disk
Need for high algorithmic parallelism

Big Data?

Most of the everyday problems are not big data.

Big Data & Machine Learning

Hadoop stack

HDFS + YARN, MapReduce, Pig, Hue

MapReduce

IPython

Highload challenges

Extreme parallelism. Each mutex is counted. epoll is essential.
Extreme storage requirements: solved by Tarantool
Tiniest memory leak becomes a disaster
- valgrind's massif is not an option sometimes
- tcmalloc heap profiling
C++ exception safety.
BL engine must be as fast as possible - LuaJIT
Logging is hard. There is a special infrastructure based on Graphite.

Examples

Bots ("autoregs")

Classify Mail registrations either as "normal" or "bot"
Problems:
- Noisy unbalanced dataset
- Variety of signatures
Features:
- Lexical
- Geolocational
- Metadata
- Other
Implementation: Python, Lua

Autoregs - algorithm

Merge different data streams
Cluster (modified s-means, similar to DBSCAN)
Classify (logistic regression)
Apply negative reinforcement
Merge with the previous state
Back by humans

Sad but true

Yeah, and one of the most important antispam tasks has always been to reduce the amount of hand work.

Sender reputation

Forecast the spam reports number
Based on sender classification and typical behavior patterns
Depends on the context, e.g. time and date
Implementation: Python, Lua

Break-ins

Detect account hijacking
Based on comparison of user's actions with the collected profile
Features:
- UI actions
- Used devices
- Geolocational
Implementation: Pig, Python

Break-ins

Secret projects

Anti-bruteforce
Fraud spam detection
First line support helper
Many, maaany more
...spam filtering, actually

Finally

I want to do machine learning

Study theory: Coursera
Study Python: sklearn
Take part in HackerRank and Kaggle challenges
Gain experience in a relevant company

To become familiar with Big Data tools, play with a single-node Hadoop installation.

I want to code highload solutions

Strong C++ is a must
Linux internals knowledge is a must
Study devops
Gain experience in a relevant company