Measuring code sentiment in a Git repository

What we want to achieve

• Run through commits in a Git repository
• For each changed file, find new comments
• Classify each comment as positive, neutral or negative
• Plot and discover!

Technologies

• src-d/go-git, src-d/hercules
• Babelfish
• Tensorflow, vmarkovtsev/BiDiSentiment
• Python, Go

src-d/go-git

• Production ready work in progress
• 2400 on GitHub
• Started in 2015 as an internal open-source project
• Highly modular, e.g. has storage backends
• Idiomatic ™

src-d/hercules

• Started in 2016 as an efficient implementation of Git-of-Theseus

• Code ownership

• Churn matrix

• Structural hotness in 3D

src-d/hercules

• Powered by the DAG abstraction with automatic dependency resolution
• Plugin system (pkg/plugin)

Babelfish

Classic approach

• Define a grammar to create parsers
• Implement them from scratch
• Good performance
• Hell of maintenance
• I am talking about you, ANTLR4

Babelfish

Our approach

• Re-use existing parsers
• Convert to the universal format
• Easy to maintain
• RPC performance
• See it in action

import (
	"gopkg.in/bblfsh/client-go.v2"
	"gopkg.in/bblfsh/client-go.v2/tools"
)
client, _ := bblfsh.NewClient("0.0.0.0:9432")
resp, _ := client.NewParseRequest().Content("...").Do()
nodes, err := tools.Filter(resp.UAST, "//*[@roleComment]")
// nodes[0].Token

Tensorflow

Tensorflow

Tensorflow

Production pipeline:

• Ensure that the model is bug-free
• Train on much data and using a GPU cluster
• Apply a metaparameter optimization algorithm
• Export to "GraphDef" format (Protocol Buffers)
• Distribute in an embedded or pluggable fashion
• Infer with any programming language

Tensorflow

• Train platform: Python, GPU
• Inference platform: Go, CPU
• Can neugram change this?

modelBytes, _ := assets.Asset("model.pb")
graph := tf.NewGraph()
graph.Import(modelBytes, "")
input := graph.Operation("input").Output(0)
output := graph.Operation("output").Output(0)
sess, _ := tf.NewSession(graph, &tf.SessionOptions{})
feed, _ := tf.NewTensor([][]float32{})
result, _ := sess.Run(
    map[tf.Output]*tf.Tensor{input: input1},
    []tf.Output{output}, nil)
		

vmarkovtsev/BiDiSentiment

• Dataset: 1.5 million tweets

vmarkovtsev/BiDiSentiment

		echo "When I wrote this, only God and I understood \
		what I was doing. Now, God only knows" | $GOPATH/bin/sentiment
		

vmarkovtsev/BiDiSentiment

		echo "sometimes I believe compiler ignores \
		all my comments" | $GOPATH/bin/sentiment
		

vmarkovtsev/BiDiSentiment

		echo "drunk, fix later" | $GOPATH/bin/sentiment
		

Slap projects together and deploy

Results

- golang/go

• It is low-level, old, and unused by Go's current HTTP stack
• not clear why this shouldn't work
• overflow causes something awful
• quiet expected TLS handshake error remote error: bad certificate

+ golang/go

• Special case, useful in debugging
• Implementation: Parallel summing of adjacent bits. See Hackers Delight, Chap. 5: Counting Bits.
• TODO(austin): This could be a really awesome string method
• TODO(bradfitz): this might be 0, once escape analysis is better

+ keras-team/keras

• With TensorFlow, we can infer the output shape directly
• Make sure there is exactly one 1 in a row
• get sorted list of layer depths
• Theano has a built-in optimization for logsumexp so we can just write the expression directly

Conclusion

• It is easy to analyse Git/code with Go
• Go is handy for inference-in-a-box
• We need a dedicated dataset for comment sentiment
• Naive sentiment alignment depends on the project type

Papers

• Classifying code comments in Java open-source software systems
• Analyzing developer sentiment in commit logs
• Sentiment analysis of commit comments in GitHub: an empirical study
• Sentiment analysis of Travis CI builds

Thank you

My contacts:

• vadim@sourced.tech
• tmarkhor
• vmarkovtsev
• blog.sourced.tech