# Descriptive Statistics

---


One of the first things a data analyst has to do after reading in a dataset, is to get a feel for the data.  A data analyst should check the size of the data, the variables, the variable names, and get summary statistics.  These help the analyst understand the data, and to ask the data generator any clarifying questions.  They may also want to investigate or take corrective measures if they find anomalies in the data.  Finally, the descriptive statistics can help set the analytical strategy.  For example, if a binary variable (eg. sex) has predominantly one category, it is of limited use in the data analysis.


## Checking data size

Checking data size can reveal if the data has been loaded correctly.  For example, if you were expecting 100 data points, and instead got 53, then you may want to find out if the data file was corrupted or there was a mistake in reading the data into R.  Let us see this via an example by reading in the Agren Arabidopsis datset.

```{r}
# Filename in string
agrenURL <- "https://raw.githubusercontent.com/sens/smalldata/master/arabidopsis/agren2013.csv"
agren <- read.csv(agrenURL)
#Print top of data frame
round(head(agren),2)
```

To see the dimensions of this data frame (spreadsheet type data) we can use `dim`.  It returns the number of rows and columns in a data frame.

```{r}
dim(agren)
```

We can also get the number of rows and the number of columns separately using `nrow` and `ncol` separately.

```{r}
nrow(agren)
```

```{r}
ncol(agren)
```

To get the variable names of a data frame (or the "names" attribute of any object) use `names`.  The variabe names are explained in the README accompanying the data.

```{r}
names(agren)
```

To get the size of a data vector, use `length`.  For example, each data column is a vector; so we can use the following.  The dollar sign is used to get a specific variable in a data frame.

```{r}
length(agren$it09)
```

## Getting type

The next thing one will want to do is to get the types of the data fields.  You can get the type of a variables using `typeof`; you can apply the function to all data columns using `lapply` as follows.

```{r}
lapply(agren,typeof)
```

This looks a bit ugly, so you may want to prettify it.

```{r}
as.matrix(sapply(agren,typeof))
```

## Summary function

After this one might want to get a better look into the actual numbers.
The `summary()` function is a handy multipurpose function in R that
provides summary data related to a dataset. The output of the summary
function depends on what type of object is input as an argument. It
usually reports what is needed in terms of summary statistics. It is a
primary function to explore how to analyze a data set.  For a data
frame, the function `summary()` returns five values: the mean, median,
25th and 75th percentiles (first and third quartiles), minimum and
maximum for each variable.

For example, let us get the summarystatistics with the function `summary()`.


In the case of a data frame, the function `summary()` is automatically applied to each variable (*i.e,* column).

```{r}
summary(agren)
```

We can calculate descriptive statistics for a specific variable (vector) as follows.

```{r}
summary(agren$it09)
```

Notice that the summary function returns the number of `NA's` or missing values.  This is also very important because missing data can be informative.  To check if there is any missing values in a dataset, we use the `anyNA` function which will just tell us if there is _any_ missing data at all.

```{r}
anyNA(agren)
```

## Basic descriptive statistics functions

R offers a number of functions to get descriptive statistics. The following table gives a list of several descriptive statistics commonly used.


|   Function name    |  Description                                                                                              |
|:-------------------|:----------------------------------------------------------------------------------------------------------|
|     `mean(x)`      |  Returns the average of the values in x                                                                    |
|     `sd(x)`        |  Returns the standard deviation of the values in x                                                        |
|     `var(x)`       |  Returns the variance of the values in x                                                                  |
|     `median(x)`    |  Returns the median of the values in x                                                                    |
|     `min(x)`       |  Returns the minimum value in x                                                                            |
|     `max(x)`       |  Returns the maximum value in x                                                                            |
|     `sum(x)`       |  Returns the total sum of all values in x                                                                  |
|     `quantile(x)`  |  Returns by default the minimum, the maximum and three quartiles (the 0.25, 0.50 and 0.75 quartiles) in x |
|     `IQR(x)`       |  Returns the interquantile range, the difference of its upper and lower quartiles in x                     |
|     `range(x)`     |  Returns the minimum and the maximum values in x                                                           |
|     `table(x)`     |  Returns a frequency table, representing the number of occurrences of every unique value in x               |



Many functions in R programming have a `na.rm` option. If we set this option to `TRUE`, observations with **NA** values will be excluded from calculations. However, if we set this option to `FALSE`, the function will return **NA** or an error if there are any missing values the dataset.

```{r}
mean(agren$it09, na.rm = FALSE)
```

```{r}
mean(agren$it09, na.rm = TRUE)
```

```{r}
quantile(agren$it09, na.rm = T)
```

The `table()` function in R  programming is especially useful to perform cross-tabulation, which summarizes categorical data.


Let us load the English Premier League dataset (2015-2016 season) which contains categorical variables such as FTR (Full Time Result) and HTR (Half Time Result) whose values belongs to the set {H=Home Win, D=Draw, A=Away Win}.

```{r}
## URL of the file
eplURL <- "https://raw.githubusercontent.com/sens/smalldata/master/soccer/E0.csv"
## read in data frame
epl <- read.csv(eplURL)
## examine head
head(epl)
```

```{r}
# Get the number of occurences for each values
table(epl$FTR)
```

The `table()` function is also helpful in generating a 2-way cross table.

```{r}
table(epl$FTR, epl$HTR, dnn = c("FTR", "HTR"))
```


## Manipulating datasets

After performing basic checks you may want to manipulate the data based on certain criteria.  For example, you may want to remove observations with impossible values.  Sometimes, it is convenient to restrict the data to individuals with complete data (no missing data).  These manipulations should be done with due care and diligence using domain-specific knowledge and input from experts.  To understand how this can be done, we need to look closer into two concepts -- logical operations and missing data representation.


## Logical (Boolean)


Logical operations are common in data analysis and programming.  For example, we may include or exclude data points based on certain criteria.  To best perform these tasks, it is helpful to consider the logical or Boolean data type.  The logical (boolean) data that has one of two possible values: `TRUE` and `FALSE`. In R programming, we can also use `T` for `TRUE` and `F` for `FALSE`.

```{r}
typeof(TRUE)
```

```{r}
typeof(T)
```

It is important to realize that `TRUE` and `FALSE` are not strings.

```{r}
typeof(FALSE)
typeof("FALSE")
```

A logical value is generally created by a boolean expression such as a comparison between variables, which returns a boolean value. The next table describes typical comparison operators

|  Operator   | Operation                                                                                                                       |
|:------------|:--------------------------------------------------------------------------------------------------------------------------------|
|      ==     | returns `TRUE` if the value on the left is equal to the value on the right, otherwise it returns `FALSE`.                      |
|      !=     | returns `TRUE` if the value on the left is different from the value on the right, otherwise it returns `FALSE`.              |
|      >      | returns `TRUE` if the value on the left is greater than the value on the right, otherwise it returns `FALSE`.              |
|      >=     | returns `TRUE` if the value on the left is greater than or equal to the value on the right, otherwise it returns `FALSE`. |
|      <      | returns `TRUE` if the value on the left is less than the value on the right, otherwise it returns `FALSE`.                      |
|      <=     | returns `TRUE` if the value on the left is less than or equal to the value on the right, otherwise it returns `FALSE`.           |



```{r}
22 < 24
```

Logical operators combine multiple boolean expressions into a single expression that returns a single logical value. The following table shows the standard logical operators.

|  Operator   | Operation                                                                                                  |
|:------------|:-----------------------------------------------------------------------------------------------------------|
|      !      | NOT returns `TRUE` if the value is `FALSE`, and returns `FALSE` if the value is `TRUE`                      |
|      &&     | AND returns `TRUE` if and only if the expressions on both sides are `TRUE`, otherwise it returns `FALSE`              |
|      &      | applies AND operation element-wise                                                                          |
|      \|\|     | OR returns `TRUE` if and only if the expression on either side is `TRUE`, otherwise it returns `FALSE`  |
|      \|      | applies OR operation element wise                                                                          |



In the case the operator `&&` compares two vectors, only the first elements of each vector are compared.

```{r}
A <- c(T, F, T, T)
B <- c(T, T, T, T)
```

```{r}
A&&B
```

```{r}
# AND acts as "Multiplication" where TRUE is 1 and FALSE is 0
A
B
A&B
```

```{r}
A==B
```

```{r}
all(A==B)
```

```{r}
# Compare A and !A
A
!A
```

```{r}
# OR acts as "Addition" where TRUE is 1 and FALSE is 0
A
B
A|B
```

Logical expressions are often necessary for subsetting data. For example, in the Arabidopsis dataset, let subset the plants whose average number of seeds is greater than 19.

```{r}
# Let subset the plants whose average number of seeds
# is greater than 19 in Italy in 2009.
agrenSub <- agren$it09[agren$it09>19]
round(head(agrenSub), 2)
```

```{r}
# Let subset the plants whose average number of seeds
# is greater than 19 in Italy in 2009, for all variables.
agrenSub <- agren[agren$it09>19,]
round(head(agrenSub), 2)
```

We will see in the missing values section how logical values can be used to handle missing data.

## Missing Data


One of the most common challenge in data analysis is how to handle missing values. In R programming, the symbol **NA** (Not Available) represents missing values. We will present various functions and options assiociated with **NA** to deal with missing data in the following sections:
* [How do we test for missing values?](#test-NA)
* [How to exclude missing values?](#exclude-NA)


Before, to go further, it is imporant to understand what is the **NA** symbol and what would be the output resulting from operations with missing data.
**NA** is  a reserved word and definined in R as a logical constant of lentgth 1. It can be generated by importing data with missing values, or it can be compelled to any data type.
Most expressions that include **NA** will generate also **NA**, with few exceptions.

```{r}
1 + NA
```

```{r}
NA^0
```

Here, the result is 1 since we could argue that any number to the power zero is equal to 1.

```{r}
FALSE || NA
```

```{r}
TRUE || NA
```

In this case, the **NA** a logical value that is missing. Therefore, it could be representing only `TRUE` or `FALSE`. Independently what **NA** represents, the answer will always be the same, `TRUE`.


### How do we test for misssing values? <a id='test-NA'></a>

We can use the two following functions `anyNA()` and `is.na()` for testing missing values.
The function `anyNA()` returns `TRUE` only if a data set contains at least one missing value, but it does not give any indication where the missing values are localized.


For example purposes, we will create a data frame with missing values.

```{r}
# Create a data frame containing NA values
df <- data.frame("ID" = c("002", "004", "006", "007", "008"),
                 "Name" = c("Bill Fairbanks", NA, "Alex Trevelyan", "James Bond", NA),
                 stringsAsFactors = FALSE)
df
```

```{r}
anyNA(df)
```

In the case, we are interested in identifying positions of the missing values we can use `is.na()`. The function `is.na()` returns `TRUE` for the elements of vector (or dataframe) that are **NA**, and `FALSE` otherwise.

```{r}
is.na(df)
```

To get the total number of **NA** we can use the function `sum()` which counts the number of `TRUE` in a data frame (or a vector).

```{r}
# Get the total number or NA in a dataframe
sum(is.na(df))
```

To obtain the exact positions of the **NA** elements, we use the function `which()` that returns the locations of all `TRUE` in a data frame (or a vector).

```{r}
# Get the positions where the elements are NA
which(is.na(df))
```

```{r}
# Get the coordinates of NA elements in a dataframe
which(is.na(df), arr.ind = TRUE)
```

Another useful function for testing missing values is `complete.cases()`. This function returns `FALSE` if a row contains **NA**, and `TRUE` otherwise.

```{r}
# Identify the rows which are complete cases (i.e. with no NA)
complete.cases(df)
```

### How to exclude misssing values? <a id='exclude-NA'></a>

There exist different ways to exclude missing values. First, we can use the `na.rm = TRUE` argument to dismiss missing values from an operation.

```{r}
# Create a vector containing NA values
vec <- c(12, 7, NA, 17, 30, 23)
vec
```

```{r}
# Get the sum of the vector vec without na.rm argument
sum(vec)
```

```{r}
# Get the sum of the vector vec with na.rm argument
sum(vec, na.rm = TRUE)
```

We can also remove missing values by creating a subset of our data with only complete observations. For example, we can use the function `complete.cases()` to obtain only the full rows of a data frame.

```{r}
# Original data frame
df
# Subset the rows
df[complete.cases(df),]
```

Finally, another simple way to deal with missing values is to eliminate all rows containing **NA** with the function `na.omit` or `na.exclude()`.

```{r}
# Original data frame
df
# Subset the rows
na.omit(df)
```

```{r}
na.exclude(df)
```

It is always essential to deal with missing values since most functions will return **NA** if they are not excluded.  The existence and pattern of missing data can also reveal information about the data generating process, and possible sources of bias.
