Public Git Archive

10x bigger than any other dataset

100x saved time

PGA in a nutshell

• 182,014 GitHub top-starred repos
• 3 TB on disk
• 54,5 million files in HEAD
• 15,9 billion lines of code in HEAD
• 455 programming languages
• 289 licenses

Language stats

Generation

1. Fetch GHTorrent
2. Threshold by 50
3. Clone with borges
4. Pack to siva
5. Index with borges-indexer

90% of repositories constitute 50% of the disk size

Getting PGA

pga list - download and list the index

pga get - download the siva files

Using PGA

1. siva unpack

   + regular Git
2. go-git
3. source{d} engine

>>> from sourced.engine import Engine
        >>> engine = Engine(spark, "/path/to/siva/files", "siva")
        >>> engine.repositories.references.head_ref \
        .commits.tree_entries.blobs \
        .classify_languages() \
        .filter('lang = "Python"') \
        .extract_uasts() \
        .query_uast('//*[@roleIdentifier]') \
        .extract_tokens("result", "tokens") \
        .select("blob_id", "path", "tokens")
        

Legal

• PGA contains personal information
• All Git data is stored in the USA
• Ability to exclude any repository from the dataset
• Index license: CC-BY-NC-SA 4.0 > ODBL

Thank you

• blog.sourced.tech
• pga.sourced.tech