Correlations and Simple Models

Data Exploration and Processing

We are not going to tell you anything about this data. This is intended to replicate a real-world experience that you will all encounter in the (possibly near) future: someone hands you data and you’re expected to make sense of it. Fortunately for us, this data is (somewhat) self-contained. We’ll first check the variable names to try to divine some information. Recall, we have a handy little function for that:

names(ameslist)

Note that, when doing data exploration, we will sometimes choose to not save our output. This is a judgement call; here we’ve chosen to merely inspect the variables rather than diving in.

Inspection yields some obvious truths. For example:

…but we face some not-so-obvious things as well. For example:

It will be difficult to learn anything about the data that is of type int without outside documentation. However, we can learn something more about the factor-type variables. In order to understand these a little better, we need to review some of the values that each take on.

Try it: Go through the variables in the dataset and make a note about your interpretation for each. Many will be obvious, but some require additional thought.

Although there are some variables that would be difficult to clean, there are a few that we can address with relative ease. Consider, for instance, the variable GarageType. This might not be that important, but, remember, the weather in Ames, IA is pretty crummy—a detached garage might be a dealbreaker for some would-be homebuyers. Let’s inspect the values:

> unique(ameslist$GarageType)
[1] Attchd  Detchd  BuiltIn CarPort <NA> Basment 2Types

With this, we could make an informed decision and create a new variable. Let’s create OutdoorGarage to indicate, say, homes that have any type of garage that requires the homeowner to walk outdoors after parking their car. (For those who aren’t familiar with different garage types, a car port is not insulated and is therefore considered outdoors. A detached garage presumably requires that the person walks outside after parking. The three other types are inside the main structure, and 2Types we can assume includes at least one attached garage of some sort). This is going to require a bit more coding and we will have to think through each step carefully.

First, let’s create a new object that has indicator variables (that is, a variable whose values are either zero or one) for each of the GarageType values. That is, it has a \(1\) if the variable takes on some specific value, and a \(0\) otherwise. Do this for all but one of the different values in GarageType, and your descriptive variable is now represented by numbers.

As with everything in R, there’s a handy function to do this for us:

GarageTemp = model.matrix( ~ GarageType - 1, data=ameslist )

We now have two separate objects living in our computer’s memory: ameslist and GarageTemp—so named to indicate that it is a temporary object.² We now need to stitch it back onto our original data; we’ll use a simple concatenation and write over our old list with the new one:

ameslist <- cbind(ameslist, GarageTemp)
> Error in data.frame(..., check.names = FALSE) :
  arguments imply differing number of rows: 1460, 1379

Huh. What’s going on?

Now that we’ve got that working (ha!) we can generate a new variable for our outdoor garage:

ameslist$GarageOutside <- ifelse(ameslist$GarageTypeDetchd == 1 | ameslist$GarageTypeCarPort == 1, 1, 0)
unique(ameslist$GarageOutside) 
[1]  0  1 NA

This seems to have worked. The command above ifelse() does what it says: if some condition is met (here, either of two variables equals one) then it returns a one; else it returns a zero. Such functions are very handy, though as mentioned above, there are other ways of doing this. Also note, that while fixed the issue with NA above, we’ve got new issues: we definitely don’t want NA outputted from this operation. Accordingly, we’re going to need to deal with it somehow.

Try it: Utilizing a similar approach to what you did above, fix this so that the only outputs are zero and one. This requires taking a stand on what the NA values mean. If you think they correspond to a detached garage (or something functionally equivalent, like “no parking whatsoever”), then change the NA values to zero. If you think they are mistakes, then we should drop all data with NA for the this column. State what you did and why. You can do this just using a subset to state which values you want to replace, or you can use case_when and make sure your last conditional always returns a value. Refresh yourself in Content 01.

Generally speaking, this is a persistent issue, and you will spend an extraordinary amount of time dealing with missing data or data that does not encode a variable exactly as you want it. This is expecially true if you deal with real-world data: you will need to learn how to handle NAs. There are a number of fixes (as always, Google is your friend) and anything that works is good. But you should spend some time thinking about this and learning at least one approach.