Take a {ghdump} to download GitHub repos

A silhouette of a dump truck dumping trash bags.

My garbage GitHub repos being dumped onto my local machine (via openclipart.org, CC0 1.0)

tl;dr

Run ghd_copy() from the {ghdump} package to either clone or download all the GitHub repositories for a given user. Intended for archival purposes or setting up a new computer.

The package comes with no guarantees and will likely be in a perpetual work-in-progress state. Please submit issues or pull requests.

Clone army

Situation:

  • Sometimes I get a new computer and want to clone all my repos to it
  • Sometimes I want to be able to archive my repos so I’m not dependent on GitHub nor any given computer
  • it would be tedious to download or clone the repos one-by-one from the GitHub interface

Wants:

  • To clone or download all of my repos with one command
  • Be able to unzip downloaded repos en masse if I want to
  • Do all this from within R, mostly for the learning experience, but also to allow for user interactivity

Observations:

  • I don’t know of a specific R function that automates mass-downloading or mass-cloning of GitHub repos
  • the {gh} package provides a lightweight GitHub API wrapper for R that’s likely to be helpful
  • R has many file-handling functions that will be helpful

{ghdump}

The result is that I wrote a function, ghd_copy(), that copies (clones or downloads) all the repos for a given user to a specified location. You can get it in the tiny {ghdump} package.

The function interacts with the GitHub API thanks to the {gh} package by Gábor Csárdi, Jenny Bryan and Hadley Wickham, while iterating over repos comes thanks to the {purrr} package by Lionel Henry and Hadley Wickham.

Get and use

Install with:

remotes::install_github("matt-dray/ghdump")

In brief, you use it like this:

ghdump::ghd_copy(
  gh_user = "matt-dray",          # download repos for this user
  dest_dir = "~/Documents/repos"  # full local file path to copy to
  copy_type = "download"          # "download" or "clone" the repos
)

This assumes you have a GitHub account and a GitHub Personal Access Token (PAT) stored in your .Renviron file. You can do this with the following steps:

usethis::browse_github_pat()  # opens browser to generate token
usethis::edit_r_environ()     # add your token to the .Renviron
# then restart R

See the Happy Git and GitHub for the useR book by Jenny Bryan, the STAT 545 TAs and Jim Hester for more info on this.

Interactivity

My expectation is to use ghd_copy() infrequently and in a non-programmatic way, so I’ve made it quite interactive. This means user input is required; you’ll get some yes/no questions in the console that will affect how the function runs.

Here’s an imaginary demo of the output from ghd_copy() when copy_type = "download":

> ghd_copy("made-up-user", "~/Desktop/test-download", "download")
Fetching GitHub repos for user made-up-user... 3 repos found
Create new directory at path ~/Desktop/test-download? y/n: y
Definitely download all 3 repos? y/n: y
Downloading zipped repositories to ~/Desktop/test-download

trying URL 'https://github.com/made-up-user/fake-repo-1/archive/master.zip'
Content type 'application/zip' length 100 bytes
==================================================
downloaded 100 bytes

trying URL 'https://github.com/made-up-user/fake-repo-2/archive/master.zip'
Content type 'application/zip' length 100 bytes
==================================================
downloaded 100 bytes

trying URL 'https://github.com/made-up-user/fake-repo-3/archive/master.zip'
Content type 'application/zip' length 100 bytes
==================================================
downloaded 100 bytes

Unzip all folders? y/n: y
Unzipping repositories
Retain the zip files? y/n: y
Keeping zipped folders.
Remove '-master' suffix from unzipped directory names? y/n: y
Renaming files to remove '-master' suffix
Finished downloading

And now imaginary demo of the output from ghd_copy() when copy_type = "clone":

> ghd_copy("mase-up-user", "~/Desktop/test-clone", "clone")
Fetching GitHub repos for user made-up-user... 3 repos found
Create new directory at path ~/Desktop/test-clone? y/n: y
Definitely clone all 3 repos? y/n: y
Cloning repositories to ~/Desktop/test-clone 
Cloning into 'fake-repo-1'...
Cloning into 'fake-repo-2'...
Cloning into 'fake-repo-3'...
Finished cloning

Note that cloning has only been tested on my own Mac OS machine at this point (June 2020) and is not guaranteed to work elsewhere yet. Please submit issues or pull requests to help improve this.

Under the hood

What are the steps to downloading repos with ghdump::ghd_copy()?

First, to get repo info:

  1. ghdump:::ghd_get_repos() passes a GitHub username to gh::gh(), which contacts the GitHub API to return a gh_response object that contains info about each of that user’s repos
  2. ghdump:::ghd_extract_names() takes the gh_response object from ghd_get_repos() and extracts the names into a character vector

Then to download (if copy_type = "download"):

  1. ghdump:::ghd_enframe_urls() turns the character vector of repo names into a data.frame, with a corresponding column that contains the URL to a zip file for that repo
  2. ghdump:::ghd_copy_zips() takes each zip file URL from that data frame and downloads them to the file path provided by the user
  3. ghdump:::ghd_unzip() unzips the zipped repos

You can, of course, use these intermediate functions if you have slightly different needs. Maybe you want to limit the repos that are downloaded; do this by filtering the vector output from ghdump:::ghd_extract_names() for example.

Or to clone (if copy_type = "clone"):

  1. ghdump:::ghd_clone_multi() that iterates cloning over the repos, itself calling ghdump:::ghd_clone_one()

Why bother?

What did I learn from doing this?

Iteration

Aside from {gh}, the package also depends on {purrr} for iterative programming.

For example, the gh_response object output from ghdump:::ghd_get_repos() is passed to map() with the pluck() function to extract the repo names.

Another example is the use of walk(), which is like map(), except we use it when the output is some ‘side effect’. By ‘side effect’, we mean that it doesn’t return an R object. For example, we can walk() the unzip() function over the path to each zip file. This doesn’t return anything in R; it results in some local files being manipulated.

File manipulation

R can be used to interact with files on your computer. There’s a number of these base R funntions in teh package:

  • dir.create() to create a new folder
  • file.remove() to remove a file or folder
  • list.files() and list.dirs() to return a character vector files and folders at some path
  • file.rename to change the name of a file or folder
  • unzip() to unpack a zipped folder

User input

How do you ask questions of your user and get answers? This interactivity is made possible by readline(). You pass it a string to prompt the user, whose return value can be stored.

For example, this is how it looks in the console:

> answer <- readline("Do you like pizza? ") 
Do you like pizza? yes
> answer
[1] "yes"

Where a user has written yes after the prompt on the second line.

Stickers

I’ve designed a few hex stickers with the {hexSticker} package; you can see them in my ‘stickers’ GitHub repo. This time I made the sticker for {ghdump} using Dmytro Perepolkin’s {bunny} package, which is a helper for the {magick} package from Jeroen Ooms. It’s a very smooth process with much flexibility.

This belongs in a dump

Yeah, maybe. It’s not sophisticated, but I’ve found it useful for my own specific purposes.