# tl;dr

I wrote a little function to check web pages for link rot and put it in the tiny R package {linkrot} in case you want to use or improve it.

You’ve clicked a link before and been taken somewhere you weren’t expecting. Sometimes it’s because you’ve been rickrolled,1 sure, but content on the internet is constantly being moved or removed and links break all the time.

A hyperlink that no longer resolves can be considered to have ‘rotted’. As time marches on, the ‘rottenness’ of the internet increases. This can be frustrating.

This blog is getting on for a hundred posts over three years. It would not be a surprise if link rot has taken hold. How big is the problem?

# Rising damp

So, basically I want to visit every link in every post on this blog and see if it’s still working.2

I’ve written the function detect_rot() to do this for any given web page and I’ve put it in the {linkrot} package on GitHub. To install:

install.packages("remotes")
library(linkrot)

In short, the detect_rot() function takes the URL of a web page and returns a tibble with details of each link from that page and whether it can be reached.

I’ve basically built it for my own amusement, so there’s no guarantees. Feel free to suggest or amend things in the GitHub repo.

## Check one post

Let’s feed in the first post on this blog, from April 2018:

trek_url <- "https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/"
trek_rot <- detect_rot(trek_url)
## Checking <https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/> ..............................

It can take a short while for the function to visit every link. To let us know it’s working, the URL is printed to the console and then a period (.) is printed for every link that’s been successfully visited (a bit like a progress bar).

We’re returned an object with a bunch of information.

str(trek_rot)
## tibble [30 × 6] (S3: tbl_df/tbl/data.frame)
##  $page : chr [1:30] "https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/" "https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/" "https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/" "https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/" ... ##$ link_url         : chr [1:30] "https://www.r-project.org/about.html" "https://en.wikipedia.org/wiki/Star_Trek:_The_Next_Generation" "http://www.st-minutiae.com/resources/scripts/#thenextgeneration" "https://github.com/zeeshanu/learn-regex/blob/master/README.md" ...
##  $link_text : chr [1:30] "R statistical software" "Star Trek: The Next Generation" "Star Trek Minutiae" "regex" ... ##$ response_code    : num [1:30] 200 200 200 200 200 200 200 404 200 200 ...
##  $response_category: chr [1:30] "Success" "Success" "Success" "Success" ... ##$ response_success : logi [1:30] TRUE TRUE TRUE TRUE TRUE TRUE ...

So, it’s a tibble with six columns and a row for each link on that page that’s been checked (30 in this example). Basically, the output tells us the URL and text of each link and also whether the page was reachable or not.

The tibble includes a special officially-standardised three-digit ‘status code’ in the response_code column. These indicate whether contact was successful, with a specific reason. For example, 200 represents a typical success (‘OK’), but you may be familiar with 404 (‘not found’) if you’ve visited a broken link before.

We can extract any broken links using the logical response_success column.

trek_rot[!trek_rot$response_success, c(4, 5, 2)] ## # A tibble: 1 x 3 ## response_code response_category link_url ## <dbl> <chr> <chr> ## 1 404 Client error https://cran.r-project.org/web/packages/rvest… So, at time of writing, that post has one broken link: an {rvest} package vignette for SelectorGadget that’s no longer active on the CRAN site. It has status code 404 (‘client error’), which basically means the thing couldn’t be found. We can confirm this by visiting the URL, but you could also use the {webshot} package to go and retrieve an screenshot of the page. library(webshot) cran_404 <- trek_rot$link_url[!trek_rot\$response_success]
webshot(cran_404, vheight = 250)

So that’s CRAN’s 404 page to tell us that the page couldn’t be fetched.

## Check whole blog

Now we know how it works for one page, we can apply the function over every post of this blog and see how many links have rotted.

First we need all the post URLs, which are all available from the blog’s homepage. The links returned are internal (like 2021/06/28/pixel-art/), so we need to add on the https://www.rostrum.blog/ bit. We also need to filter out any links that aren’t posts (like the ‘About’ page).

# Load packages
library(xml2)
library(rvest)
library(dplyr)
library(purrr)

# The URL of this blog's homepage
blog_url <- "https://www.rostrum.blog"

blog_hrefs <-
read_html(blog_url) %>%  # get full homepage HTML
html_nodes("a") %>%      # nodes with links, <a>
html_attr("href")        # the URL attribute

posts <- paste0(blog_url, blog_hrefs[grepl("^/20", blog_hrefs)])
tail(posts)  # preview
## [1] "https://www.rostrum.blog/2018/06/05/tid-ye-text/"
## [2] "https://www.rostrum.blog/2018/05/25/cloud-pie/"
## [3] "https://www.rostrum.blog/2018/05/19/pokeballs-in-super-smash-bros/"
## [4] "https://www.rostrum.blog/2018/05/12/accessibility-workshop-at-sprint18/"
## [6] "https://www.rostrum.blog/2018/04/14/r-trek-exploring-stardates/"

Now we can use {purrr} to iterate the detect_rot() function over the pages. By using map_df() we can get a data frame as output rather than a list. I’ve hidden the printed output from detect_rot() this time because there would be nearly 100 lines of output (one per post).

results <- map_df(posts, detect_rot)

Again, we can filter the logical response_success column to see which links weren’t successfully resolved.

rotten <- filter(results, !response_success)
nrow(rotten)
## [1] 61

So in total there were 61 links out of 2331 that did not return a ‘success’, which works out to about 3% being unreachable.

We can count the reasons for these failures by looking at the status codes.

count(rotten, response_code, sort = TRUE)
## # A tibble: 6 x 2
##   response_code     n
##           <dbl> <int>
## 1           404    53
## 2           400     4
## 3           403     1
## 4           406     1
## 5           410     1
## 6           502     1

You can see most of these status codes are in the 4xx range, which is the group of codes that mean ‘client error’. Usually this is a problem with the link you’ve provided, like 404 is ‘not found’, 403 is ‘forbidden’ and 406 is ‘not acceptable’.

It’s hard to tell whether this level of link rot is good or bad, but remember that these are links that have failed within the past three years. Imagine how bad this might look in another 10 years. By comparison, a quarter of links on the New York Times website were completely inaccessible, stretching back to 1996.

I’d be interested to know whether this is comparable to your blog or website.

# Surveying for rot

We’ve seen it in action, but how does the function work? I’m not claiming the approach is optimal, but it obviously worked for my needs. You’ll probably find the approach naive if you have any experience in dealing with HTTP requests from R.

## Validate, fetch, check

You can find the function definition for detect_rot() in the {linkrot} source code. It has three underlying steps, each of which has a helper function:

1. Check that the provided URL is valid with .validate_page()
2. Scrape the links from the page with .fetch_links()
3. Visit each link and check its response code with .check_links()

So, the URL provided by the user is first checked with help from the {httr} package. We GET() the page and then extract the status_code() and check for an http_error(). If all is well (i.e. no error), then we can continue.

To get the links from the URL, we first scrape the page with xml2::read_html() and then use {rvest} functions: html_nodes() to grab all the nodes with links, then html_attr() and html_text() to extract the URLs and link text from each.

Finally, each of the URLs is visited with GET() and the http_status() is extracted. The final data frame is converted to tibble (for ease of reading) and returned to the user.

## Limitations

Of course, it’s possible that GET() will fail to reach a page for reasons other than it being missing. Sometimes there can be a momentary blip, but detect_rot() is simple and never retries a link.

Additionally, there are some links that {httr} struggles to contact. I wrapped functions internal to detect_rot() inside tryCatch() so any failures appear as NA in the response_code column. The printed output for detect_rot() also displays an exclamation point (!) instead of a period (.) when being run. For example, there were 8 links that had this problem for this blog.

I welcome any thoughts or suggestions, particularly around testing. I’d like to use this package as a way to learn proper HTTP testing and have found rOpenSci’s HTTP Testing in R book useful so far. Eventually I might convert detect_rot() to use the {httr2} package when it’s released.

# Now what?

I could go back and fix the broken links, but maybe it’s not that big a deal. I don’t have any data on what people click on, so I can’t really tell if it’s worth it.

But anyway, didn’t I say ‘decay is inevitable’? I can fix things, but more things will break.

I wasn’t expecting this to get quite so existential.4

1. Don’t worry, you can see from the URL that this doesn’t go to the YouTube video! it goes to the excellent pudding.cool site, which has some great analysis of the rise and rise (and rise) of Rickrolling.↩︎

2. That’s links to other pages on the internet, because links also exist to take you to these footnotes, or point elsewhere internally to this website.↩︎

3. This number may be larger than 93 (the number of posts before this one) if I’ve re-knitted this post after its publication.↩︎

4. Actually yes, dear reader, he was; he really was.↩︎