Scalable Language-Agnostic Analysis of Source Code and VCS History

Vadim Markovtsev
source{d}

Huge codebases

	LoC, 10⁶
Chrome	20
Windows 10	50
Facebook	70
Eclipse Foundation	160
Apache Foundation	190
Google	2,000
GitHub	>56,000

Problems

Discover repositories
Gather them together → dataset
Run simple analysis
Run advanced analysis (e.g., ML-driven)

source{d} solutions

Discover repositories
Gather them together → dataset
Run simple analysis
Run advanced analysis (e.g., ML-driven)

borges

Features

Does scalable code retrieval
Can work with HDFS
Open source

siva

Archive format similar to TAR
Low-cost append
Constant-time random file access
Effective git push inside

Challenges

Git packfiles require efficient seek()
HDFS does not provide efficient seek()
Git repositories are unbalanced
Metadata must be stored separately

Engine

Features

Language-agnostic code and Git analysis
... with SQL syntax - gitbase
... with GUI
Provides APIs
Open source

Easy to try

           $ srcd init /path/to/git/repos
           $ srcd web sql

Top 10 repositories by commit count

SELECT r.repository_id, COUNT(*) AS commit_count
FROM ref_commits AS r
WHERE r.ref_name = 'HEAD'
GROUP BY r.repository_id
ORDER BY commit_count DESC LIMIT 10

Examples

Any Git statistics
cloc
Get specific files
REGEXP over file contents at each revision
Query code

How to parse

Regular expressions - Pygments, highlight.js
Abstract syntax tree (AST) - ANTLR
Compilation

Universal AST

± uniform structure across languages
± standard node types (roles)
XPath queries
4 traversal orders

            $ srcd web parse

Number of functions per Go file

SELECT files.repository_id, files.file_path,
    ARRAY_LENGTH(uast_extract(UAST(
        files.blob_content,
        LANGUAGE(files.file_path, files.blob_content),
        '//*[@roleFunction and @roleDeclaration]'), 'token')
    ) as functions
FROM files NATURAL JOIN ref_commits AS rc
WHERE rc.ref_name = 'HEAD' AND rc.history_index = 0
      AND LANGUAGE(files.file_path, files.blob_content) = 'Go'

Challenges

Logical planning of the queries
Performance optimization
Indexes
Caches
Efficient loading Git data from disk

Hercules

Features

Advanced Git analysis CLI
Integrated with the ecosystem
Open source

Line burndown

git blame foo.py

●2014

class Foo:
  def bar(self):
    print("!")

●2015

class Foo:
  def bar(self):
    print("?")
 
  def baz(self):
    print("!")

●2016

class FooBarBaz:
  def bar(self):
    """..."""
    print("yo")
 
  def baz(self):
    print("!")

Line burndown

Linux

Reproduce

            hercules --burndown --first-parent --pb \
    git://github.com/torvalds/linux | \
    labours.py -m project -f pb

Line overwrite matrix

Reproduce

            hercules --burndown --burndown-people --pb \
    git://github.com/tensorflow/tensorflow | \
    labours.py -m churn_matrix -f pb

Reproduce

            hercules --couples --pb \
    git://github.com/tensorflow/tensorflow | \
    labours.py -m couples -f pb

It is also possible to embed developers, functions and classes.

Summary

borges + Engine = 💪 code/Git analysis platform
Hercules gains 👍 insights from Git
All of those are open source and easy to 👐
📚 docs.sourced.tech/engine
👀 the live demo later today!

Thank you

bit.ly/2QMOY0Q