# 2 The Basics of the R Programming Language

Most of the work in this book is implemented in the R statistical programming language which, along with Python, is one of the two languages that I use in my day-to-day statistical analysis. Sample implementations in Python are also provided at various points in the book, as well as implementations in Julia for those who are keen on this newer and up-and-coming language. I have made efforts to keep the code as simple as possible, and I have tried to avoid the use of too many external packages. For the most part, readers should see (especially in the earlier chapters) that code blocks are short and simple, relying wherever possible on base R functionality. No doubt there are neater and more effective ways to code some of the material in this book using a wide array of R packages—and some of these are illustrated in Chapter 10—but my priority has been to keep the code simple, consistent and easily reproducible.

For those who wish to follow the method and theory without the implementations in this book, there is no need to read this chapter. However, the style of this book is to use implementation to illustrate theory and practice, and so tolerance of many code blocks will be necessary as you read onward.

For those who wish to simply replicate the models as quickly as possible, they will be able to avail of the code block copying feature, which appears whenever you scroll over an input code block. Assuming all the required external packages have been installed, these code blocks should all be transportable and immediately usable. For those who are extra-inquisitive and want to explore how I constructed graphics used for illustration (for which code is usually not displayed), the best place to go is the Github repository for this book.

This chapter is for those who wish to learn the methods in this book but do not know how to use R. However, it is not intended to be a full tutorial on R. There are many more qualified individuals and existing resources that would better serve that purpose—in particular I recommend . It is recommended that you consult these resources and become comfortable with the basics of R before proceeding into the later chapters of this book. However, acknowledging that many will want to dive in sooner rather than later, this chapter covers the absolute basics of R that will allow the uninitiated reader to proceed with at least some orientation.

## 2.1 What is R?

R is a programming language that was originally developed by and for statisticians, but in recent years its capabilities and the environments in which it is used have expanded greatly, with extensive use nowadays in academia and the public and private sectors. There are many advantages to using a programming language like R. Here are some:

1. It is completely free and open source.
2. It is faster and more efficient with memory than popular graphical user interface analytics tools.
3. It facilitates easier replication of analysis from person to person compared with many alternatives.
4. It has a large and growing global community of active users.
5. It has a large and rapidly growing universe of packages, which are all free and which provide the ability to do an extremely wide range of general and highly specialized tasks, statistical and otherwise.

There is often heated debate about which tools are better for doing non-trivial statistical analysis. I personally find that R provides the widest array of resources for those interested in inferential modeling, while Python has a more well-developed toolkit for predictive modeling and machine learning. Since the primary focus of this book is inferential modeling, the in-depth walkthroughs are coded in R.

## 2.2 How to start using R

Just like most programming languages, R itself is an interpreter which receives input and returns output. It is not very easy to use without an IDE. An IDE is an Integrated Development Environment, which is a convenient user interface allowing an R programmer to do all their main tasks including writing and running R code, saving files, viewing data and plots, integrating code into documents and many other things. By far the most popular IDE for R is RStudio. An example of what the RStudio IDE looks like can be seen in Figure 2.1.

To start using R, follow these steps:

2. Download the latest version of the RStudio IDE from https://rstudio.com/products/rstudio/ and view the video on that page to familiarize yourself with its features.
3. Open RStudio and play around.

The initial stages of using R can be challenging, mostly due to the need to become familiar with how R understands, stores and processes data. Extensive trial and error is a learning necessity. Perseverance is important in these early stages, as well as an openness to seek help from others either in person or via online forums.

## 2.3 Data in R

As you start to do tasks involving data in R, you will generally want to store the things you create so that you can refer to them later. Simply calculating something does not store it in R. For example, a simple calculation like this can be performed easily:

3 + 3
## [1] 6

However, as soon as the calculation is complete, it is forgotten by R because the result hasn’t been assigned anywhere. To store something in your R session, you will assign it a name using the <- operator. So I can assign my previous calculation to an object called my_sum, and this allows me to access the value at any time.

# store the result
my_sum <- 3 + 3

# now I can work with it
my_sum + 3
## [1] 9

You will see above that you can comment your code by simply adding a # to the start of a line to ensure that the line is ignored by the interpreter.

Note that assignment to an object does not result in the value being displayed. To display the value, the name of the object must be typed, the print() command used or the command should be wrapped in parentheses.

# show me the value of my_sum
my_sum
## [1] 6
# assign my_sum + 3 to new_sum and show its value
(new_sum <- my_sum + 3)
## [1] 9

### 2.3.1 Data types

All data in R has an associated type, to reflect the wide range of data that R is able to work with. The typeof() function can be used to see the type of a single scalar value. Let’s look at the most common scalar data types.

Numeric data can be in integer form or double (decimal) form.

# integers can be signified by adding an 'L' to the end
my_integer <- 1L
my_double <- 6.38

typeof(my_integer)
## [1] "integer"
typeof(my_double)
## [1] "double"

Character data is text data surrounded by single or double quotes.

my_character <- "THIS IS TEXT"
typeof(my_character)
## [1] "character"

Logical data takes the form TRUE or FALSE.

my_logical <- TRUE
typeof(my_logical)
## [1] "logical"

### 2.3.2 Homogeneous data structures

Vectors are one-dimensional structures containing data of the same type and are notated by using c(). The type of the vector can also be viewed using the typeof() function, but the str() function can be used to display both the contents of the vector and its type.

my_double_vector <- c(2.3, 6.8, 4.5, 65, 6)
str(my_double_vector)
##  num [1:5] 2.3 6.8 4.5 65 6

Categorical data—which takes only a finite number of possible values—can be stored as a factor vector to make it easier to perform grouping and manipulation.

categories <- factor(
c("A", "B", "C", "A", "C")
)

str(categories)
##  Factor w/ 3 levels "A","B","C": 1 2 3 1 3

If needed, the factors can be given order.

# character vector
ranking <- c("Medium", "High", "Low")
str(ranking)
##  chr [1:3] "Medium" "High" "Low"
# turn it into an ordered factor
ranking_factors <- ordered(
ranking, levels = c("Low", "Medium", "High")
)

str(ranking_factors)
##  Ord.factor w/ 3 levels "Low"<"Medium"<..: 2 3 1

The number of elements in a vector can be seen using the length() function.

length(categories)
## [1] 5

Simple numeric sequence vectors can be created using shorthand notation.

(my_sequence <- 1:10)
##  [1]  1  2  3  4  5  6  7  8  9 10

If you try to mix data types inside a vector, it will usually result in type coercion, where one or more of the types are forced into a different type to ensure homogeneity. Often this means the vector will become a character vector.

# numeric sequence vector
vec <- 1:5
str(vec)
##  int [1:5] 1 2 3 4 5
# create a new vector containing vec and the character "hello"
new_vec <- c(vec, "hello")

# numeric values have been coerced into their character equivalents
str(new_vec)
##  chr [1:6] "1" "2" "3" "4" "5" "hello"

But sometimes logical or factor types will be coerced to numeric.

# attempt a mixed logical and numeric
mix <- c(TRUE, 6)

# logical has been converted to binary numeric (TRUE = 1)
str(mix)
##  num [1:2] 1 6
# try to add a numeric to our previous categories factor vector
new_categories <- c(categories, 1)

# categories have been coerced to background integer representations
str(new_categories)
##  num [1:6] 1 2 3 1 3 1

Matrices are two-dimensional data structures of the same type and are built from a vector by defining the number of rows and columns. Data is read into the matrix down the columns, starting left and moving right. Matrices are rarely used for non-numeric data types.

# create a 2x2 matrix with the first four integers
(m <- matrix(c(1, 2, 3, 4), nrow = 2, ncol = 2))
##      [,1] [,2]
## [1,]    1    3
## [2,]    2    4

Arrays are n-dimensional data structures with the same data type and are not used extensively by most R users.

### 2.3.3 Heterogeneous data structures

Lists are one-dimensional data structures that can take data of any type.

my_list <- list(6, TRUE, "hello")
str(my_list)
## List of 3
##  $: num 6 ##$ : logi TRUE
##  $: chr "hello" List elements can be any data type and any dimension. Each element can be given a name. new_list <- list( scalar = 6, vector = c("Hello", "Goodbye"), matrix = matrix(1:4, nrow = 2, ncol = 2) ) str(new_list) ## List of 3 ##$ scalar: num 6
##  $vector: chr [1:2] "Hello" "Goodbye" ##$ matrix: int [1:2, 1:2] 1 2 3 4

Named list elements can be accessed by using $. new_list$matrix
##      [,1] [,2]
## [1,]    1    3
## [2,]    2    4

Dataframes are the most used data structure in R; they are effectively a named list of vectors of the same length, with each vector as a column. As such, a dataframe is very similar in nature to a typical database table or spreadsheet.

# two vectors of different types but same length
names <- c("John", "Ayesha")
ages <- c(31, 24)

# create a dataframe
(df <- data.frame(names, ages))
##    names ages
## 1   John   31
## 2 Ayesha   24
# get types of columns
str(df)
## 'data.frame':    2 obs. of  2 variables:
##  $names: chr "John" "Ayesha" ##$ ages : num  31 24
# get dimensions of df
dim(df)
## [1] 2 2

## 2.4 Working with dataframes

The dataframe is the most common data structure used by analysts in R, due to its similarity to data tables found in databases and spreadsheets. We will work almost entirely with dataframes in this book, so let’s get to know them.

To work with data in R, you usually need to pull it in from an outside source into a dataframe3. R facilitates numerous ways of importing data from simple .csv files, from Excel files, from online sources or from databases. Let’s load a data set that we will use later—the salespeople data set, which contains some information on the sales, average customer ratings and performance ratings of salespeople. The read.csv() function can accept a URL address of the file if it is online.

# url of data set
url <- "http://peopleanalytics-regression-book.org/data/salespeople.csv"

# load the data set and store it as a dataframe called salespeople
salespeople <- read.csv(url)

We might not want to display this entire data set before knowing how big it is. We can view the dimensions, and if it is too big to display, we can use the head() function to display just the first few rows.

dim(salespeople)
## [1] 351   4
# hundreds of rows, so view first few
head(salespeople)
##   promoted sales customer_rate performance
## 1        0   594          3.94           2
## 2        0   446          4.06           3
## 3        1   674          3.83           4
## 4        0   525          3.62           2
## 5        1   657          4.40           3
## 6        1   918          4.54           2

We can view a specific column by using $, and we can use square brackets to view a specific entry. For example if we wanted to see the 6th entry of the sales column: salespeople$sales[6]
## [1] 918

Alternatively, we can use a [row, column] index to get a specific entry in the dataframe.

salespeople[34, 4]
## [1] 3

We can take a look at the data types using str().

str(salespeople)
## 'data.frame':    351 obs. of  4 variables:
##  $promoted : int 0 0 1 0 1 1 0 0 0 0 ... ##$ sales        : int  594 446 674 525 657 918 318 364 342 387 ...
##  $customer_rate: num 3.94 4.06 3.83 3.62 4.4 4.54 3.09 4.89 3.74 3 ... ##$ performance  : int  2 3 4 2 3 2 3 1 3 3 ...

We can also see a statistical summary of each column using summary(), which tells us various statistics depending on the type of the column.

summary(salespeople)
##     promoted          sales       customer_rate    performance
##  Min.   :0.0000   Min.   :151.0   Min.   :1.000   Min.   :1.0
##  1st Qu.:0.0000   1st Qu.:389.2   1st Qu.:3.000   1st Qu.:2.0
##  Median :0.0000   Median :475.0   Median :3.620   Median :3.0
##  Mean   :0.3219   Mean   :527.0   Mean   :3.608   Mean   :2.5
##  3rd Qu.:1.0000   3rd Qu.:667.2   3rd Qu.:4.290   3rd Qu.:3.0
##  Max.   :1.0000   Max.   :945.0   Max.   :5.000   Max.   :4.0
##                   NA's   :1       NA's   :1       NA's   :1

Note that there is missing data in this dataframe, indicated by NAs in the summary. Missing data is identified by a special NA value in R. This should not be confused with "NA", which is simply a character string. The function is.na() will look at all values in a vector or dataframe and return TRUE or FALSE based on whether they are NA or not. By adding these up using the sum() function, it will take TRUE as 1 and FALSE as 0, which effectively provides a count of missing data.

sum(is.na(salespeople))
## [1] 3

This is a small number of NAs given the dimensions of our data set and we might want to remove the rows of data that contain NAs. The easiest way is to use the complete.cases() function, which identifies the rows that have no NAs, and then we can select those rows from the dataframe based on that condition. Note that you can overwrite objects with the same name in R.

salespeople <- salespeople[complete.cases(salespeople), ]

# confirm no NAs
sum(is.na(salespeople))
## [1] 0

We can see the unique values of a vector or column using the unique() function.

unique(salespeople$performance) ## [1] 2 3 4 1 If we need to change the type of a column in a dataframe, we can use the as.numeric(), as.character(), as.logical() or as.factor() functions. For example, given that there are only four unique values for the performance column, we may want to convert it to a factor. salespeople$performance <- as.factor(salespeople$performance) str(salespeople) ## 'data.frame': 350 obs. of 4 variables: ##$ promoted     : int  0 0 1 0 1 1 0 0 0 0 ...
##  $sales : int 594 446 674 525 657 918 318 364 342 387 ... ##$ customer_rate: num  3.94 4.06 3.83 3.62 4.4 4.54 3.09 4.89 3.74 3 ...
subset(subset = salespeople$sales < 500) %>% # get the subsetted data mean() %>% # take the mean value round() # round to the nearest integer ## [1] 389 The pipe operator is unique to R and is very widely used because it helps to make code more readable, it reduces complexity, and it helps orient around a common ‘grammar’ for the manipulation of data. The pipe operator helps you structure your code more clearly around nouns (objects), verbs (functions) and adverbs (arguments of functions). One of the most developed sets of packages in R that follows these principles is the tidyverse family of packages, which I encourage you to explore. ## 2.6 Errors, warnings and messages As I mentioned earlier in this chapter, getting familiar with R can be frustrating at the beginning if you have never programmed before. You can expect to regularly see messages, warnings or errors in response to your commands. I encourage you to regard these as your friend rather than your enemy. It is very tempting to take the latter approach when you are starting out, but over time I hope you will appreciate some wisdom from my words. Errors are serious problems which usually result in the halting of your code and a failure to return your requested output. They usually come with an indication of the source of the error, and these can sometimes be easy to understand and sometimes frustratingly vague and abstract. For example, an easy-to-understand error is: subset(salespeople, subset = sales = 720) Error: unexpected '=' in "subset(salespeople, subset = sales =" This helps you see that you have used sales = 720 as a condition to subset your data, when you should have used sales == 720 for precise equality. A much more challenging error to understand is: head[salespeople] Error in head[salespeople] : object of type 'closure' is not subsettable When first faced with an error that you can’t understand, try not to get frustrated and proceed in the knowledge that it usually can be fixed easily and quickly. Often the problem is much more obvious than you think, and if not, there is still a 99% likelihood that others have made this error and you can read about it online. The first step is to take a look at your code to see if you can spot what you did wrong. In this case, you may see that you have used square brackets [] instead of parentheses () when calling your head() function. If you cannot see what is wrong, the next step is to ask a colleague or do an internet search with the text of the error message you receive, or to consult online forums like https://stackoverflow.com. The more experienced you become, the easier it is to interpret error messages. Warnings are less serious and usually alert you to something that you might be overlooking and which could indicate a problem with the output. In many cases you can ignore warnings, but sometimes they are an important reminder to go back and edit your code. For example, you may run a model which doesn’t converge, and while this does not stop R from returning results, it is also very useful for you to know that it didn’t converge. Messages are pieces of information that may or may not be useful to you at a particular point in time. Sometimes you will receive messages when you load a package from your library. Sometimes messages will keep you up to date on the progress of a process that is taking a long time to execute. ## 2.7 Plotting and graphing As you might expect in a well-developed programming language, there are numerous ways to plot and graph information in R. If you are doing exploratory data analysis on fairly simple data and you don’t need to worry about pretty appearance or formatting, the built-in plot capabilities of base R are fine. If you need a pretty appearance, more precision, color coding or even 3D graphics or animation, there are also specialized plotting and graphing packages for these purposes. In general when working interactively in RStudio, graphical output will be rendered in the Plots pane, where you can copy it or save it as an image. ### 2.7.1 Plotting in base R The simplest plot function in base R is plot(). This performs basic X-Y plotting. As an example, this code will generate a scatter plot of customer_rate against sales in the salespeople data set, with the results displayed in Figure 2.3. Note the use of the arguments main, xlab and ylab for customizing the axis labels and title for the plot. # scatter plot of customer_rate against sales plot(x = salespeople$sales, y = salespeople$customer_rate, xlab = "Sales ($m)", ylab = "Average customer rating",
main = "Scatterplot of Sales vs Customer Rating")

Histograms of data can be generated using the hist() function. This command will generate a histogram of performance as displayed in Figure 2.4. Note the use of breaks to customize how the bars appear.

# convert performance ratings back to numeric data type for histogram
salespeople$performance <- as.numeric(salespeople$performance)

# histogram of performance ratings
hist(salespeople$performance, breaks = 0:4, xlab = "Performance Rating", main = "Histogram of Performance Ratings") Box and whisker plots are excellent ways to see the distribution of a variable, and can be grouped against another variable to see bivariate patterns. For example, this command will show a box and whisker plot of sales grouped against performance, with the output shown in Figure 2.5. Note the use of the formula and data notation here to define the variable we are interested in and how we want it grouped. We will study this formula notation in greater depth later in this book. # box plot of sales by performance rating boxplot(formula = sales ~ performance, data = salespeople, xlab = "Performance Rating", ylab = "Sales ($m)",
main = "Boxplot of Sales by Performance Rating")

These are among the most common plots used for data exploration purposes. They are examples of a wider range of plotting and graphing functions available in base R, such as line plots, bar plots and other varieties which you may see later in this book.

### 2.7.2 Specialist plotting and graphing packages

By far the most commonly used specialist plotting and graphing package in R is ggplot2. ggplot2 allows the flexible construction of a very wide range of charts and graphs, but uses a very specific command grammar which can take some getting used to. However, once learned, ggplot2 can be an extremely powerful tool. Many of the illustratory figures used in this book are developed using ggplot2 and while the code for these figures is generally not included for the sake of brevity, you can always find it in the source code of this book on Github6. A great learning resource for ggplot2 is .

The plotly package allows the use of the plotly graphing library in R. This is an excellent package for interactive graphing and is used for 3D illustrations in this book. Output can be rendered in HTML—allowing the user to play with and explore the graphs interactively—or can be saved as static 2D images.

GGally is a package that extends ggplot2 to allow easy combination of charts and graphs. This is particularly valuable for quicker exploratory data analysis. One of its most popular functions is ggpairs(), which produces a pairplot. A pairplot is a visualization of all univariate and bivariate patterns in a data set, with univariate distributions in the diagonal and bivariate relationships or correlations displayed in the off-diagonal. Figure 2.6 is an example of a pairplot for the salespeople data set, which we will explore further in Chapter 5.

library(GGally)

# convert performance and promotion to categorical
salespeople$promoted <- as.factor(salespeople$promoted)
salespeople$performance <- as.factor(salespeople$performance)

# pairplot of salespeople
GGally::ggpairs(salespeople)

## 2.8 Documenting your work using R Markdown

For anyone performing any sort of multivariate analysis using a statistical programming language, appropriate documentation and reproducibility of the work is essential to its success and longevity. If your code is not easily obtained or run by others, it is likely to have a very limited impact and lifetime. Learning how to create integrated documents that contain both text and code is critical to providing access to your code and narration of your work.

R Markdown is a package which allows you to create integrated documents containing both formatted text and executed code. It is, in my opinion, one of the best resources available currently for this purpose. This entire book has been created using R Markdown. You can start an R Markdown document in RStudio by installing the rmarkdown package and then opening a new R Markdown document file, which will have the suffix .Rmd.

R Markdown documents always start with a particular heading type called a YAML header, which contains overall information on the document you are creating. Care must be taken with the precise formatting of the YAML header, as it is sensitive to spacing and indentation. Usually a basic YAML header is created for you in RStudio when you start a new .Rmd file. Here is an example.

---
title: "My new document"
author: "Keith McNulty"
date: "25/01/2021"
output: html_document
---

The output part of this header has numerous options, but the most commonly used are html_document, which generates your document as a web page, and pdf_document, which generates your document as a PDF using the open source LaTeX software package. If you wish to create PDF documents you will need to have a version of LaTeX installed on your system. One R package that can do this for you easily is the tinytex package. The function install_tinytex() from this package will install a minimal version of LaTeX which is fine for most purposes.

R Markdown allows you to build a formatted document using many shorthand formatting commands. Here are a few examples of how to format headings and place web links or images in your document:

# My top heading

![](path/to/image)

Code can be written and executed and the results displayed inline using backticks. For example, writing

r nrow(salespeople)

inline will display 351 in the final document. Entire code blocks can be included and executed by using triple-backticks. The following code block:

{r}
# show the first few rows of salespeople


will display this output:

##   promoted sales customer_rate performance
## 1        0   594          3.94           2
## 2        0   446          4.06           3
## 3        1   674          3.83           4
## 4        0   525          3.62           2
## 5        1   657          4.40           3
## 6        1   918          4.54           2

The {} wrapping allows you to specify different languages for your code chunk. For example, if you wanted to run Python code instead of R code you can use {python}. It also allows you to set options for the code chunk display separated by commas. For example, if you want the results of your code to be displayed, but without the code itself being displayed, you can use {r, echo = FALSE}.

The process of compiling your R Markdown code to produce a document is known as ‘knitting.’ To create a knitted document, you simply need to click on the ‘Knit’ button in RStudio that appears above your R Markdown code.

If you are not familiar with R Markdown, I strongly encourage you to learn it alongside R and to challenge yourself to write up any practice exercises you take on in this book using R Markdown. Useful cheat sheets and reference guides for R Markdown formatting and commands are available through the Cheatsheets section of the Help menu in RStudio. I also recommend for a really thorough instruction and reference guide.

## 2.9 Learning exercises

### 2.9.1 Discussion questions

1. Describe the following data types: numeric, character, logical, factor.
2. Why is a vector known as a homogeneous data structure?
3. Give an example of a heterogeneous data structure in R.
4. What is the difference between NA and "NA"?
5. What operator is used to return named elements of a list and named columns of a dataframe?
6. Describe some functions that are used to manipulate dataframes.
7. What is a package and how do you install and use a new package?
8. Describe what is meant by ‘namespacing’ and why it might be useful.
9. What is the pipe operator, and why is it popular in R?
10. What is the difference between an error and a warning in R?
11. Name some simple plotting functions in base R.
12. Name some common specialist plotting and graphing packages in R.
13. What is R Markdown, and why is it useful to someone performing analysis using programming languages?

### 2.9.2 Data exercises

1. Create a character vector called my_names that contains all your first, middle and last names as elements. Calculate the length of my_names.
2. Create a second numeric vector called which which corresponds to my_names. The entries should be the position of each name in the order of your full name. Verify that it has the same length as my_names.
3. Create a dataframe called names, which consists of the two vectors my_names and which as columns. Calculate the dimensions of names.
4. Create a new dataframe new_names with the which column converted to character type. Verify that your command worked using str().
5. Load the ugtests data set via the peopleanalyticsdata package or download it from the internet7. Calculate the dimensions of ugtests and view the first three rows only.
6. View a statistical summary of all of the columns of ugtests. Determine if there are any missing values.
7. View the subset of ugtests for values of Yr1 greater than 50.
8. Install and load the package dplyr. Look up the help for the filter() function in this package and try to use it to repeat the task in the previous question.
9. Write code to find the mean of the Yr1 test scores for all those who achieved Yr3 test scores greater than 100. Round this mean to the nearest integer.
10. Familiarize yourself with the two functions filter() and pull() from dplyr. Use these functions to try to do the same calculation in the previous question using a single unbroken piped command. Be sure to namespace where necessary.
11. Create a scatter plot using the ugtests data with Final scores on the $$y$$ axis and Yr3 scores on the $$x$$ axis.
12. Create your own 5-level grading logic and use it to create a new finalgrade column in the ugtests data set with grades 1–5 of increasing attainment based on the Final score in ugtests. Generate a histogram of this finalgrade column.
13. Using your new ugtests data with the extra column from the previous exercise, create a box plot of Yr3 scores grouped by finalgrade.
14. Knit all of your answers to these exercises into an R Markdown document. Create one version that displays your code and answers, and another that just displays the answers.