2 Data Preparation and Cleaning

2.1 Introduction to Clinical Data Structures

Clinical research generates various data types with specific characteristics and challenges. Understanding these structures is essential for effective data preparation.

2.1.1 Common Clinical Data Sources

Clinical data typically originates from:

Electronic Data Capture (EDC) systems: Purpose-built for clinical trials
Electronic Health Records (EHR): Patient records from healthcare systems
Patient-reported outcomes (ePRO): Data collected directly from patients
Laboratory data: Standardized clinical measurements
Wearable devices: Continuous monitoring data
Imaging data: Radiological and other imaging outputs

2.1.2 Standard Data Formats in Clinical Research

Clinical research often uses standardized data formats:

Code

# Example of importing CDISC SDTM formatted data
library(haven)
library(tidyverse)

# Import SAS dataset with SDTM structure
demographics <- read_sas("data/dm.sas7bdat")
vitals <- read_sas("data/vs.sas7bdat") 
labs <- read_sas("data/lb.sas7bdat")

# View the structure
glimpse(demographics)

The most common standardized formats include:

CDISC SDTM (Study Data Tabulation Model): Standardized structure for submission data
CDISC ADaM (Analysis Data Model): Datasets optimized for analysis
OMOP (Observational Medical Outcomes Partnership): Common data model for observational studies
FHIR (Fast Healthcare Interoperability Resources): Standard for healthcare data exchange

For more detailed information on CDISC standards and their implementation in R, see Chapter ?sec-cdisc.

# Data Preparation and Cleaning

## Introduction to Clinical Data Structures

Clinical research generates various data types with specific characteristics and challenges. Understanding these structures is essential for effective data preparation.

### Common Clinical Data Sources

```{r}
#| echo: false
#| fig-cap: "Typical Clinical Data Sources and Formats"

library(ggplot2)
library(dplyr)

# Example visualization of clinical data sources
data_sources <- tibble(
  source = c("EDC Systems", "EHR/EMR", "eCOA/ePRO", "Labs", "Wearables", "Imaging"),
  importance = c(90, 85, 75, 80, 60, 70)
)

ggplot(data_sources, aes(x = reorder(source, importance), y = importance)) +
  geom_col(fill = "#0072B2") +
  coord_flip() +
  labs(x = "Data Source", y = "Relative Prevalence (%)",
       title = "Common Clinical Data Sources") +
  theme_minimal() +
  theme(plot.title = element_text(face = "bold", hjust = 0.5))
```

Clinical data typically originates from:

1. **Electronic Data Capture (EDC) systems**: Purpose-built for clinical trials
2. **Electronic Health Records (EHR)**: Patient records from healthcare systems
3. **Patient-reported outcomes (ePRO)**: Data collected directly from patients
4. **Laboratory data**: Standardized clinical measurements
5. **Wearable devices**: Continuous monitoring data
6. **Imaging data**: Radiological and other imaging outputs

### Standard Data Formats in Clinical Research

Clinical research often uses standardized data formats:

```{r}
#| echo: true
#| eval: false

# Example of importing CDISC SDTM formatted data
library(haven)
library(tidyverse)

# Import SAS dataset with SDTM structure
demographics <- read_sas("data/dm.sas7bdat")
vitals <- read_sas("data/vs.sas7bdat") 
labs <- read_sas("data/lb.sas7bdat")

# View the structure
glimpse(demographics)
```

The most common standardized formats include:

- **CDISC SDTM** (Study Data Tabulation Model): Standardized structure for submission data
- **CDISC ADaM** (Analysis Data Model): Datasets optimized for analysis
- **OMOP** (Observational Medical Outcomes Partnership): Common data model for observational studies
- **FHIR** (Fast Healthcare Interoperability Resources): Standard for healthcare data exchange

For more detailed information on CDISC standards and their implementation in R, see Chapter @sec-cdisc.