This is a bit of a diversion for a public-health-focused course in data science, but it’s fun, related to web data, strings, and factors, and emphasizes tools in data wrangling. It’s most closely related to content in the Data Wrangling II topic.


Learning Objectives

Use tidytext to organize text data, and to conduct frequency and sentiment analyses.

Slide Deck

Video Lecture


I’ll write code for today’s content in a new R Markdown document called tidy_text.Rmd, and put it in the extra topics directory / GitHub repo. I’m going to load the tidyverse as usual, as well as tidytext and rvest.


## Loading required package: xml2
## Attaching package: 'rvest'
## The following object is masked from 'package:purrr':
##     pluck
## The following object is masked from 'package:readr':
##     guess_encoding


We’re sticking with “Napoleon Dynamite reviews”! First, I’ll re-use code from iteration and listcols to scrape the 1000 most recent reviews on Amazon (and cache the result).

read_page_reviews <- function(url) {
  html = read_html(url)
  review_titles = 
    html %>%
    html_nodes(".a-text-bold span") %>%
  review_stars = 
    html %>%
    html_nodes("#cm_cr-review_list .review-rating") %>%
    html_text() %>%
    str_extract("^\\d") %>%
  review_text = 
    html %>%
    html_nodes(".review-text-content span") %>%
    html_text() %>% 
    str_replace_all("\n", "") %>% 
    title = review_titles,
    stars = review_stars,
    text = review_text

url_base = ""

dynamite_reviews = 
    page = 1:100,
    urls = str_c(url_base, page)) %>% 
  mutate(reviews = map(urls, read_page_reviews)) %>% 
  unnest(reviews) %>%
  mutate(review_num = row_number()) %>% 
  relocate(page, review_num)

The output of the code above is a successfully scraped dataset with 6 and 1000 rows – one row for each review. For each review we get the title of that review, the number of stars it received, and text that describers the users feelings about the movie.

Words and wordcounts

To illustrate tidy text and text analysis, we’ll focus on the reviews directly, which are stored as strings in text. To begin our analysis, we’ll un-nest the tokens (i.e. words) in each row; the result is a tidy dataset in which each word is contained within a separate row.

dynamite_words = 
  dynamite_reviews %>% 
  unnest_tokens(word, text)

There are lots of words here that are uninformative. We’ll remove “stop words” using anti_join; in other settings the words you want to remove might be different.


dynamite_words = 
  anti_join(dynamite_words, stop_words)
## Joining, by = "word"

Great! Let’s take a look at the most commonly used (informative) words in this dataset.

dynamite_words %>% 
  count(word, sort = TRUE) %>% 
  top_n(10) %>% 
  mutate(word = fct_reorder(word, n)) %>% 
  ggplot(aes(x = word, y = n)) + 
  geom_bar(stat = "identity", fill = "blue", alpha = .6) + 
## Selecting by n

Comparing words across groups

The next code chunk below produces a table of the most frequently used in one- and five-star reviews.

dynamite_words %>%
  filter(stars %in% c(1, 5)) %>%
  group_by(stars) %>%
  count(word) %>% 
  top_n(5) %>%
## Selecting by n
stars word n
1 bad 7
1 dumb 8
1 dvd 7
1 movie 50
1 time 9
1 watch 8
5 classic 102
5 funny 125
5 love 138
5 movie 430
5 time 99

The table above gives the top 5 most frequently used words in 1-star and 5-star reviews. Movie is the most used word for both 1 and 5-star reviews, though other words, like dumb differentiate 1-star reviews from 5-star reviews, which have words like love.

Word frequency might be misleading because there are 812 5-star reviews and only 69 1-star reviews.

Let’s compare which words are more likely to come from a 1 versus 5 star ratings. We limit to words that appear at least 5 times and compute the approximate log odds ratio for each word.

word_ratios = 
  dynamite_words %>%
  filter(stars %in% c(1, 5)) %>% 
  count(word, stars) %>%
  group_by(word) %>% 
  filter(sum(n) >= 5) %>%
  ungroup() %>%
    names_from = stars, 
    values_from = n,
    names_prefix = "stars_",
    values_fill = 0) %>%
    stars_1_odds = (stars_1 + 1) / (sum(stars_1) + 1),
    stars_5_odds = (stars_5 + 1) / (sum(stars_5) + 1),
    log_OR = log(stars_5_odds / stars_1_odds)
  ) %>%

Next, let’s plot the top 10 most distinctive words (that is, words that appear much more frequently in one group than the other) below.

word_ratios %>% 
  mutate(pos_log_OR = ifelse(log_OR > 0, "5 star > 1 star", "1 star > 5 star")) %>%
  group_by(pos_log_OR) %>%
  top_n(10, abs(log_OR)) %>%
  ungroup() %>%
  mutate(word = fct_reorder(word, log_OR)) %>%
  ggplot(aes(word, log_OR, fill = pos_log_OR)) +
  geom_col() +
  coord_flip() +
  ylab("log odds ratio (5/1)") +
  scale_fill_discrete(name = "")

Words like “classic”, “awesome”, and “love” have high relative frequency in the 5-star reviews and “boring”, “dumb”, and “bad” have high relative frequency in the 1-star reviews. This seems to be a polarizing film.

Sentiment analysis

Finally, let’s score the sentiment in each word. We’ll use the “bing” (like Bing Liu, not like sentiment lexicon, which simply categorizes each word as having a positive or negative sentiment.

bing_sentiments = get_sentiments("bing")

Note this is might not always be appropriate – this scores cold as negative which might not be accurate for e.g. food inspections – but we’ll use it anyway.

We need to combine this lexicon with our tidy dataset containing words from each inspection. Note that only words that are in the sentiment lexicon will be retained, as the rest of the words are not considered meaningful. We’ll also count the number of positive and negative words in each review, and create a score that is the difference between the number of positive words and negative words.

dynamite_sentiments = 
  dynamite_words %>% 
  inner_join(., bing_sentiments) %>% 
  count(review_num, sentiment) %>% 
    names_from = sentiment, 
    values_from = n, 
    values_fill = 0) %>% 
  mutate(review_sentiment = positive - negative) %>% 
  select(review_num, review_sentiment)
## Joining, by = "word"

We now have sentiment scores for each inspection. We’ll combine these with our original dataset, which had inspections in each row rather than words in each row – the data tidied for text analysis aren’t really suitable for our current needs.

dynamite_sentiments = 
    dynamite_reviews, dynamite_sentiments, 
    by = "review_num")

Finally, let’s make a plot showing inspection sentiments and stars.

dynamite_sentiments %>% 
    review_num = factor(review_num),
    review_num = fct_reorder(review_num, review_sentiment, .desc = TRUE)) %>% 
  ggplot(aes(x = review_num, y = review_sentiment, fill = stars, color = stars)) + 
  geom_bar(stat = "identity") + 

Sentiment seems to be at least somewhat associated with star rating in that more positive sentiments are more yellow-green (4-5 stars) and more negative sentiments are more blue-purple.

Here is the text from the most positive review:

dynamite_sentiments %>%
  filter(review_sentiment == max(review_sentiment)) %>% 
## [1] "love love love love love. We watch this twice a week and even had a Napoleon Dynamite party complete with corn dogs and tater tots. FUN!!"

And here is the text from the most negative (1-star) review:

dynamite_sentiments %>%
  filter(review_sentiment == min(review_sentiment), stars == 1) %>% 
## [1] "Let me start by openly admitting that I am an 80's throwback and proud of it. I understand that my era did not produce horribly intelligent, deeply touching movies. That being said, this movie puts the Du-h! in Dumb!!  My teenage daughter had seen it a sleepover, so we rented it so she could show me how silly it is for our girly night.  She actually got tickled at a few spots on her second run through it. I sat open mouthed, amazed at the sheer senselessness of the whole thing...waiting desperately for a plot or growth of a character. None came. Since we had it rented for the weekend, she showed it to her older brother the next night. She giggled again, and he snickered with a confused look on his face once in a while. Then we couldn't let my Hubby miss out on the sheer stupidity of it all, so we watched it with him on Sunday afternoon.  Our daughter and son both snickered here and there, mostly at the looks on their parents' faces.  My husband was, as was I, gobsmackered. We kept looking at each other as if to assure each other that it was indeed that painfully dumb.  The only real emotion I felt was a horribly heartbroken, sick to my stomach feeling for Napoleon and his friends for being so awkward and socially inept...I remember kids who struggled in school that way, and I always almost cried every time I saw them in a tough situation.  If anything it brought back bad memories of school....who needs that?  Anyway, all this to say it is a story about an awkward boy who uses bluster and dumb responses to deal with not fitting in.  It follows (if you can exaggerate and say there is a plot) him as he finds friends (just as goofy as he) and finds romance...sort of."

Other materials

  • The framework we used is explained in detail in the Tidy Text book
  • One of the book’s authors, Julia Silge, has a nice video talking about the work
  • The other of the book’s authors, Dave Robinson, used the approach to examine Donald Trump’s tweets in this this blog post

The code that I produced working examples in lecture is here.