# Natural Language Processing¶

Please refer to the video lecture for the full information on this topic and code set-up.

In this lecture we will show an example of how you could use R to perform Natural Language Processing. There is no assignment or project attached to this topic because this topic can range widely. You should also note that a lot of times R is not the best choice for NLP, and that other languages such as Python are a stronger choice due to their library support.

If you are looking for a supplemental assignment, read through the great walkthrough on NLP written here.

## Install the necessary libraries¶

We'll need the following libraries:

- tm
- twitteR
- wordcloud
- RColorBrewer
- e1017
- class

You can install them with this code (uncomment it first):

In [54]:

    #install.packages('tm',repos='http://cran.us.r-project.org')
    #install.packages('twitteR',repos='http://cran.us.r-project.org')
    #install.packages('wordcloud',repos='http://cran.us.r-project.org')
    #install.packages('RColorBrewer',repos='http://cran.us.r-project.org')
    #install.packages('e1017',repos='http://cran.us.r-project.org')
    #install.packages('class',repos='http://cran.us.r-project.org')

# Create a Twitter App¶

This project requires you to create a twitter account and a twitter application if you want to follow along. Let's outline the steps to do this:

1. Create an Account on Twitter
2. Create a new app at: https://apps.twitter.com/
3. You may need to point it to a personal URL, in which case you may need to create a wordpress site or something similar
4. Get Your Keys Under the Keys and Access Tokens tab

5. Then use them with the twitteR library:

        getTwitterOAuth(consumer_key, consumer_secret)

## Regular Expression Review¶

Now let's review a few key Regular Expression functions we've touched upon earlier:

### grep()¶

Return the index location of pattern matches

In [9]:

    args(grep)

Out[9]:

    function (pattern, x, ignore.case = FALSE, perl = FALSE, value = FALSE,
        fixed = FALSE, useBytes = FALSE, invert = FALSE)
    NULL

In [13]:

    grep('A', c('A','B','C','D','A'))

Out[13]:

1. 1
2. 5

### nchar()¶

length of a string

In [14]:

    args(nchar)

Out[14]:

    function (x, type = "chars", allowNA = FALSE, keepNA = FALSE)
    NULL

In [17]:

    nchar('helloworld')

Out[17]:
10
In [18]:

    nchar('hello world')

Out[18]:
11

### gsub()¶

perform replacement of the matching patterns

In [19]:

    args(gsub)

Out[19]:

    function (pattern, replacement, x, ignore.case = FALSE, perl = FALSE,
        fixed = FALSE, useBytes = FALSE)
    NULL

In [20]:

    gsub('pattern','replacement','hello have you seen the pattern here?')

Out[20]:
'hello have you seen the replacement here?'

## Text Manipulation¶

### paste()¶

concatenate several strings together

In [32]:

    print(paste('A','B','C',sep='...'))

    [1] "A...B...C"

In [30]:

    #help(paste)

### substr()¶

returns the substring in the given character range start:stop for the given

In [33]:

    substr('abcdefg',start=2,stop = 5)

Out[33]:
'bcde'

### strsplit()¶

splits a string into a list of substrings based on another string split in x

In [49]:

    strsplit('2016-01-23',split='-')

Out[49]:

1. 1. '2016'
   2. '01'
   3. '23'

# NLP Important Terms and Concepts¶

- Document - The individual text document (e.g. a resume)
- Corpus - The collection of documents (e.g. A group of resumes)
- Bag-of-Words - unordered collection of words (e.g. list of unordered words)
- n-grams - contiguous sequence of n items from a given sequence of text (e.g. ['A','G','C',T']
- Stopwords - words that appear too often to be of great importance (e.g. the,a,I,etc...)
- Tokens - Any combination of characters (words)
- Stemming - Process to remove suffixes of words (e.g. run,runner,running all reduce to base word - run)
- TF-IDF : term frequency-inver document frequency is a statistic that tells how important a word is in a given corpus, its a way of determining high-information words
- Term Document Matrix - representation of a document collection as vectors

# Twitter Mining¶

Let's mine twitter for some general data!

## Step 1: Import Libraries¶

In [ ]:

    library(twitteR)
    library(tm)
    library(wordcloud)
    library(RColorBrewer)

## Step 2: Search for Topic on Twitter¶

We'll use the twitteR library to data mine twitter. First you need to connect by setting up your Authorization keys and tokens.

In [ ]:

    setup_twitter_oauth(consumer_key, consumer_secret, access_token=NULL, access_secret=NULL)

We will search twitter for the term 'soccer'

In [ ]:

    soccer.tweets <- searchTwitter("soccer", n=2000, lang="en")
    soccer.text <- sapply(soccer.tweets, function(x) x$getText())

## Step 3: Clean Text Data¶

We'll remove emoticons and create a corpus

In [ ]:

    soccer.text <- iconv(soccer.text, 'UTF-8', 'ASCII') # remove emoticons
    soccer.corpus <- Corpus(VectorSource(soccer.text)) # create a corpus

## Step 4: Create a Document Term Matrix¶

We'll apply some transformations using the TermDocumentMatrix Function

In [ ]:

    term.doc.matrix <- TermDocumentMatrix(soccer.corpus,
                                          control = list(removePunctuation = TRUE,
                                                         stopwords = c("soccer","http", stopwords("english")),
                                                         removeNumbers = TRUE,tolower = TRUE))

## Step 5: Check out Matrix¶

In [ ]:

    head(term.doc.matrix)

In [ ]:

    term.doc.matrix <- as.matrix(term.doc.matrix)

## Step 6: Get Word Counts¶

In [ ]:

    word.freqs <- sort(rowSums(term.doc.matrix), decreasing=TRUE)
    dm <- data.frame(word=names(word.freqs), freq=word.freqs)

## Step 7: Create Word Cloud¶

In [ ]:

    wordcloud(dm$word, dm$freq, random.order=FALSE, colors=brewer.pal(8, "Dark2"))

# Great Job!¶