Discrete Data and Continuous Data

Discrete Data, as the name suggests, can take only specified values. For example, when you roll a die, the possible outcomes are 1, 2, 3, 4, 5 or 6 and not 1.5 or 2.45 or when you toss a coin, the possible outcomes are either head or tail.

Continuous Data can take any value within a given range. The range may be finite or infinite. For example, A girl’s weight or height, the length of the road. The weight of a girl can be any value from 54 kgs, or 54.5 kgs, or 54.5436kgs. It can be in Point or fractions.

Train and test Data sets formation using R

Before building any model for machine learning, there is one of the golden rules of machine learning and modelling in general: models are built using training data, and evaluated on testing data. The reason is overfitting: most models’ accuracy can be artificially increased to a point where they “learn” every single detail of the data used to build them; unfortunately, it usually means they lose the capability to generalise. That’s why we need unseen data (i.e., the testing set): if we overfit the training data, the performance on the testing data will be poor. In real life, simple models often beat complex ones, because they can generalise much better. We will do a random 70:30 split in our data set (70% will be for training models, 30% to evaluate them). For reproducibility, we will need to set the seed of the random number generator (it means every time I run the code, I’ll get the same train and test sets. Here’s  goes the code:

> # Reproducing same set; 222 has no particular meaning, just taken 
> set.seed(222) 

> # randomly pick 70% of the number of observations 400
> data<- sample(1:nrow(mydata),size = 0.7*nrow(mydata)) 

> # subset mydata to include only the elements in the data
> train <- mydata[data,] 

> # subset mydata to include all but the elements in the data i.e. 30%
> test <- mydata[-data,] 

> nrow(train)
[1] 280 

> nrow(test)
[1] 120

You can use library(ggplot2) to plot the train and test data by creating dataframe and plot it.

Ensemble Predictions : Combine Model Predictions Into Ensemble Predictions

Many times happens that it take too much time to find well performing machine learning algorithms for your dataset. Trial and error nature of applied machine learning is the reason behind it.

Once we have a selected list of accurate models, we can use algorithm tuning to get the most from each algorithm.

Another approach that we can use to increase accuracy on our dataset is to combine the predictions of multiple different models together.

Combine Model Predictions Into Ensemble Predictions

The three most popular methods for combining the predictions from different models are:

·         Bagging. Building multiple models (typically of the same type) from different subsamples of the training dataset.

·         Boosting. Building multiple models (typically of the same type) each of which learns to fix the prediction errors of a prior model in the chain.

·         Stacking. Building multiple models (typically of differing types) and supervisor model that learns how to best combine the predictions of the primary models.

Sentiment Analysis in R with package syuzhet

Sentiment analysis is the process of determining whether a piece of writing  or set of text is positive, negative or neutral. Here, we’ll work with the package “syuzhet”.

Supposed there is long email which we would like to check, we’ll read the Emails from the database.

Read emails into syuzhet
1 2	Emails <- data.frame(dbGetQuery(database,"SELECT * FROM Emails")) library('syuzhet')

“syuzhet” uses NRC Emotion lexicon.

The NRC emotion lexicon is a list of words and their associations with eight emotions (trust, surprise, anger, fear, anticipation, sadness, joy, and disgust) and two sentiments (negative and positive).

The get_nrc_sentiment function returns a data frame in which each row represents a sentence from the original file. The columns include one for each emotion type was well as the positive or negative sentiment valence. It allows us to take a body of text and return which emotions it represents — and also whether the emotion is positive or negative. 

Do sentiment analysis of the email
1 2 3 4 5 6 7 8 9 10 11	d<-get_nrc_sentiment(Emails$RawText) td<-data.frame(t(d)) td_new <- data.frame(rowSums(td[2:7945])) #The function rowSums computes column sums across rows for each level of a grouping variable. #Transformation and  cleaning names(td_new)[1] <- "count" td_new <- cbind("sentiment" = rownames(td_new), td_new) rownames(td_new) <- NULL td_new2<-td_new[1:8,]

Now, we’ll use “ggplot2” to create a bar graph. Each bar represents how prominent the each of the emotion is in text.

Graph the sentiment analysis in ggplot2
1 2 3	#Visualisation library("ggplot2") qplot(sentiment, data=td_new2, weight=count, geom="bar",fill=sentiment)+ggtitle("Email sentiments")

Generating a wordcloud of any text - Text Mining in R

Load the below library, if already installed.
If required to install then use the below code:

install.packages("wordcloud")

install.packages("RColorBrewer")

# Generate the WordCloud

library("wordcloud")

library("RColorBrewer")

par(bg="grey30")

png(file="WordCloud.png",width=1000,height=700, bg="grey30") # file shall save in your default directory

wordcloud(d$word, d$freq, col=terrain.colors(length(d$word), alpha=0.9), random.order=FALSE, rot.per=0.3 )

title(main = "Your effort", font.main = 1, col.main = "cornsilk3", cex.main = 1.5)

dev.off()

Text preprocessing for Text Mining in R (stemming)

While analyzing text, we need to preprocess it. Text data contains white spaces, punctuations, stop words etc. These characters do not convey much information and are hard to process. For example, English stop words like "an", “the”, “is”, "are" etc. do not tell you much information about the sentiment of the text, entities mentioned in the text, or relationships between those entities. Depending upon the task at hand, we deal with such characters differently. This will help isolate text mining in R on important words. 

• Convert the text to lower case, so that words like “wrong” and “Wrong” are considered the same word for analysis
• Remove numbers
  
• Remove English stopwords e.g “are”, “is”, “of”, etc
• Remove punctuation e.g “,”, “?”, etc
• Eliminate extra white spaces
• Stemming text 

Stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form. E.g changing “laptop”, “laptops”, “laptop’s”, “laptops’” to “laptop”. This can also help with different verb tenses with the same semantic meaning such as see, saw, and seen. 

One very useful library to perform the above steps and text mining in R is the “tm” package. The main structure for managing documents in tm is called a Corpus, which represents a collection of text documents.

Cleaning text in R

# Transform and clean the text library("tm") docs <- Corpus(VectorSource(textdata)) Using the TM library to process text # Convert the text to lower case docs <- tm_map(docs, content_transformer(tolower)) # Remove numbers docs <- tm_map(docs, removeNumbers) # Remove english common stopwords docs <- tm_map(docs, removeWords, stopwords("english")) # Remove punctuations docs <- tm_map(docs, removePunctuation) # Eliminate extra white spaces docs <- tm_map(docs, stripWhitespace) To stem text, we will need another library also, known as SnowballC which will shared in my another blog.

How to look at the structure of the dataframe in R

str function gives the structure of dataframe with type of data for all variables and data sample also. Data type can be number, integer, char, factor etc.

str(my_data)