Machine Learning on Source Code

#MLonCode

Vadim Markovtsev

@vadimlearning

🚀 Machine Learning

GitHub

Machine Learning on Source Code

MLonCode

Examples

Projects similar to MariaDB/server?

Details

Similar code detection

By style
By structure
By identifiers

provides us

💪Global graph
Licenses
Refactoring

Another example: DeepCode.ai.

class foobar:
    def connecttoserver(self):
        myserverhost = globalconfig.server.host

class FooBar:
    def connect_to_server(self):
        myServerHost = globalConfig.server.host

Prediction of class, function, variable names
Type inference
Which comments do not make sense?
Which comments are funny?
Which APIs are bad or misused?

Your code as a crime scene

Idea: mine the development history
Book by Adam Tornhill
codescene.io

Tools for MLonCode

Data

Datasets

GHTorrent - everything except Git repositories, 70GB
Public Git Archive - only Git repositories, 3TB
GitHub Data in Google BigQuery
rovers & borges - DIY

PGA

270k of siva files
CSV index

Details in the paper.

Plumbing

source{d} engine

siva loader for Apache Spark
Classification, parsing, scaling
Apache license

GitHub

source{d} engine

>>> from sourced.engine import Engine
>>> engine = Engine(spark, "/path/to/siva/files", "siva")
>>> engine.repositories.references.head_ref \
    .commits.tree_entries.blobs \
    .classify_languages() \
    .select("blob_id", "path", "lang") \
    .show()

How to parse

Regular expressions - Pygments, highlight.js
Abstract syntax tree (AST) - ANTLR
Compilation

AST

Universal AST

± uniform structure
± standard node types (roles)
XPath queries
4 traversal orders

dashboard.bblf.sh

>>> engine.repositories.references.head_ref \
    .commits.tree_entries.blobs \
    .classify_languages() \
    .filter('lang = "Python"') \
    .extract_uasts() \
    .query_uast('//*[@roleIdentifier]') \
    .extract_tokens("result", "tokens") \
    .select("blob_id", "path", "tokens")

Powerful

modelforge

Automatically fetch from the internet
"Model Store"
Flexible, modern binary format - ASDF
Versioning
Reproducability
Support various programming languages

GitHub

Machine Learning

hercules

Go
Mining Git history
Relies on go-git, Babelfish and Tensorflow
Apache license

GitHub

MLonCode in hercules

Structural hotness
Comment sentiment

source{d} ml

Python
Targets large-scale MLonCode
Runs on top of source{d} engine, modelforge and Tensorflow
Apache license

GitHub

Solved practical problems

Identifier splitting
O(1) code similarity search
Topic modeling
Embeddings

Identifier embeddings

$ \begin{split} V_1 \Leftrightarrow & \,\texttt{"foo"} \\ \\ V_2 \Leftrightarrow & \,\texttt{"bar"} \\ \\ V_3 \Leftrightarrow & \,\texttt{"integrate"} \end{split} $

$ distance(V_1, V_2) < distance(V_1, V_3) $

$$ distance(V_i, V_j) = \arccos \frac{V_i \cdot V_j}{\left\lVert V_i \right\rVert \left\lVert V_j \right\rVert} $$

$$ V_i \cdot V_j = PMI_{ij} = \log\frac{C_{ij} \sum C}{\sum_{k = 1}^N C_{ik}\sum_{k = 1}^N C_{jk}} $$

Watch the talk on dotAI 2018

Nearest to “foo”

afoo
qux
myfoo
baz
mfoo
wibble
dofoo
quux
dfoo
testing
ifoo

Analogies

“bug” - “test” + “expect” = “suppress”

“database” - “query” + “tune” = “settings”

“send” - “receive” + “pop” = “push”

Typos

recieve = receive
grey = gray
calback = callbak = callback

>>> from sourced.ml.models import Id2Vec
>>> m = Id2Vec().load()
>>> print(m.embeddings[m["foo"]])

Summary

MLonCode is fun
There is data
There are tools
Community is forming

Idea ? Doubt ? Write us !

Thank you

bit.ly/2JOBSj3