Experience Sampling Data-Wrangling

Merging datasets

Merging datasets is often essential after ESM data collection given questionnaires sent out during the ESM phase are frequently flanked by intake and outtake questionnaires (e.g., to measure age, gender, etc.). To examine relations between variables measured from these datasets all variables must be in the same dataset. Merging datasets is therefore a necessity.

So how would I go about merging the below (sample) ESM phase dataset…

…with the below (sample) intake questionnaire dataset?

A simple solution is to use the ‘merge’ function of base R. In the code below, ‘x’ refers to the ESM data, ‘y’ refers to the intake data, and ‘all.x = TRUE’ ensures the ESM data will not be lost upon merging. The variable ‘PID’ (i.e., participant identification number) is present in both datasets and is the key that connects them.

datX3 <- merge(x=datX, y=datX2, by="PID", all.x = TRUE)

Removing columns and rows via set criteria

The need to remove rows and columns/variables is commonplace in most research. However, it is particularly commonplace in ESM research where numerous variables are often measured for exploratory purposes and won’t make it into the final polished dataset. Also, ESM approaches can be prone to producing poor quality data that will ultimately need removing to find a signal amidst the error.

Lets say I want to remove a column from the table above - say, participants reported bodyweight at the beginning of the data collection period (i.e., the variable ‘WeightT1’). The below code would suffice for this.

datX3$WeightT1 <- NULL 

If I want to remove rows from a dataframe, I will need a different approach because removal will generally be based upon set criteria. Thus, if I want to remove rows where the variable ‘NumCEpi’ is ‘0’, I would instead opt for using the below code.

datX5 <-datX4[!(datX4$NumCEpi == 0),]

Remove duplicate data

It often results in ESM research that participants complete more than one questionnaire from a questionnaire link sent to them. Generally this is not wanted, thus this duplicate data will need removing. You can do this using the ‘duplicated’ function of base R.

For instance, if you scroll to the bottom of the table below, you will see participant ‘72420025120’ completed the third questionnaire sent to them on day 5 of the ESM phase twice (i.e., for this participant two rows contain DAY = 5 and SIG = 3).