Introduction

My name is Evgeni and I am a mathematician with strong passion for data analysis and R.

The first experience in the field came to me in the early 2013, in the middle of my university graduate year. I decided to take a “Data Analysis” course at Coursera (taught by amazing Jeff Leek) which later transformed into Data Science Specialization from John Hopkins University. After watching some videos, it came to me that knowledge of some weird programming language called R is needed in order to complete the course. That insight led me to another Coursera course “Computing for Data Analysis” (offered by no less amazing Roger D. Peng). Thanks to them, I fell in love with data analysis and R, which really helped me to get a wonderful job as mathematician-analyst after graduation.

Years went by until autumn 2016 (about a year ago), when I suddenly discovered the R for Data Science book and other works by Hadley Wickham. At the time I had heard only about ggplot2 but nothing more, as base R was pretty enough for my work. After this discovery I fell into a rabbit hole: tidyverse, R Markdown, reproducible research, writing R packages and so on. Finally, after blogdown reached CRAN, I realized that it is time to give back to the great R community by being more web present. Basically it means creating and maintaining own blog/site.

Goal

Being perfectionist, I spent quite some time figuring out the name of my future site. In ideal world it should have the next features:

• Be representative of the site content.
• Be memorable to audience. In general it means to have a pleasant pronunciation, to be quite short and to evoke colourful images in people’s minds.
• Be available as domain name and twitter account.

I had some possible versions of the name, but none of them actually felt right. Eventually, I came up with an idea of giving tribute to three persons mentioned above for sharing their knowledge with the world (and me in particular). The goal was to take one book authored by each of them and perform some text analysis in order to get inspiration. This also felt like a good opportunity to try tidytext package.

Next books were chosen (which had big influence on me):

• The Elements of Data Analytic Style (EDAS) by Jeff Leek.
• The Art of Data Science (ADS) by Roger D. Peng and Elizabeth Matsui.
• R for Data Science (R4DS) by Garrett Grolemund and Hadley Wickham.

I highly recommend everybody interested in data analysis to read these books. All of them can be obtained for free (but I strongly encourage to reward authors): EDAS and ADS as pdf files, R4DS as bookdown repository from Hadley’s github.

Tidy pdf

Here is the code of function for tidying pdf file at the file path. At first, it, with help of pdftools package, converts pdf file into list of strings (text per page). Then this list is converted into tibble with the following structure:

• page <int> : Word’s page number.
• line <int> : Word’s line number on page (empty lines are ignored).
• word <chr> : Word.

tidy_pdf <- function(file) {
  pages_list <- file %>%
    pdftools::pdf_text() %>%
    lapply(. %>% strsplit("\n") %>% `[[`(1))

  lapply(seq_along(pages_list), function(i) {
    page_lines <- pages_list[[i]]
    n_lines <- length(page_lines)

    tibble(
      page = rep(i, n_lines),
      line = seq_len(n_lines),
      text = page_lines
    ) %>%
      tidytext::unnest_tokens(word, text, token = "words")
  }) %>%
    bind_rows()
}

This function is enough to scrape text from both EDAS and ADS.

Tidy Rmd

The R4DS book can be obtained as a collection of Rmd files. The goal is to extract text, not code or metadata. Unfortunately, I didn’t find an easy way to do that with knitr or rmarkdown, so I wrote function myself. The tidy_rmd() reads file line by line at location file and collapses them into one string. After that, with some magic of regular expressions and str_replace_all() from stringr, it removes YAML header, code and LaTeX blocks. Note that this function is designed to only handle Rmd files with relatively simple formatting and can return undesirable output in some edge cases.

The output is a tibble with the following structure:

• name <int> : Document name which is given as function argument. Default is file’s base name without extension.
• word <chr> : Word.

tidy_rmd <- function(file, name = file_base_name(file)) {
  file_string <- file %>%
    readLines() %>%
    paste0(collapse = " ") %>%
    # Remove YAML header
    str_replace_all("^--- .*?--- ", "") %>%
    # Remove code
    str_replace_all("```.*?```", "") %>%
    str_replace_all("`.*?`", "") %>%
    # Remove LaTeX
    str_replace_all("[^\\\\]\\$\\$.*?[^\\\\]\\$\\$", "") %>%
    str_replace_all("[^\\\\]\\$.*?[^\\\\]\\$", "")

  tibble(name = name, text = file_string) %>%
    tidytext::unnest_tokens(word, text, token = "words")
}

file_base_name <- function(file) {
  file %>%
    basename() %>%
    str_replace_all("\\..*?$", "")
}

Filter good words

Only meaningful words should be used in text analysis. They are:

• Not stop words. tidytext package has a data frame stop_words with the most common stop words.
• Words that contain only alphabetic characters. This feature is, of course, arguable. I decided to add it because otherwise data can contain some links and words with ' : “we’ll”, “word’s” etc. Moreover, this is not very strict project, so I think it is appropriate decision.

Words scraped from Rmd files should be processed a little more carefully: they can contain emphasis characters “_" and “*" at the beginning or end. So, before filtering good words, these symbols should be removed.

remove_md_emphasis <- function(x) {
  str_replace_all(x, "^[_*]+|[_*]+$", "")
}

filter_good_words <- function(word_tbl) {
  word_tbl %>%
    anti_join(tidytext::stop_words, by = "word") %>%
    filter(str_detect(word, pattern = "^[[:alpha:]]+$"))
}

filter_good_words() should be applied to data frames with character column word.

Scrape books

prepare_book <- function(book_name) {
  # File should be in the data-raw folder of working directory
  file.path("data-raw", paste0(book_name, ".pdf")) %>%
    tidy_pdf() %>%
    filter_good_words() %>%
    mutate(
      id = seq_len(n()),
      book = rep(book_name, n())
    ) %>%
    select(id, book, everything())
}

edas <- prepare_book("EDAS")
ads <- prepare_book("ADS")

# Get file names used to knit the book
# r4ds folder should be in the data-raw folder of working directory
r4ds_file_names <- readLines(file.path("data-raw", "r4ds", "_bookdown.yml")) %>%
  paste0(collapse = " ") %>%
  str_extract_all('\\".*\\.[Rr]md\\"') %>%
  `[[`(1) %>%
  str_replace_all('"', '') %>%
  str_split(",[:space:]*") %>%
  `[[`(1)

# Orginize book data into tibble
r4ds_pages <- tibble(
  page = seq_len(length(r4ds_file_names)),
  file = r4ds_file_names,
  pageName = file_base_name(r4ds_file_names)
)

# Scrape book
r4ds <- file.path("data-raw", "r4ds", r4ds_pages[["file"]]) %>%
  lapply(tidy_rmd) %>%
  bind_rows() %>%
  rename(pageName = name) %>%
  # Remove md emphasis before filtering words with only alphabetic characters
  mutate(word = remove_md_emphasis(word)) %>%
  filter_good_words() %>%
  mutate(
    id = seq_len(n()),
    book = rep("R4DS", n())
  ) %>%
  left_join(y = r4ds_pages %>% select(page, pageName),
            by = "pageName") %>%
  select(id, book, page, pageName, word)