Given a messy student performance dataset with missing scores, inconsistent date formats, and duplicate records, perform end-to-end data cleaning and generate summary statistics using Pandas.
Data preprocessing accounts for up to 80% of data science effort. A robust cleaning pipeline addresses four primary anomalies: duplicates, format inconsistencies, missing values, and date parsing.
For numerical columns with skew, the median is resilient against extreme outliers compared to the arithmetic mean.
import pandas as pd
import numpy as np
# Create representative messy DataFrame
raw_data = {
"student_id": [101, 102, 103, 104, 104, 105],
"subject": ["Python", "python", "Nursing", "Law", "Law", np.nan],
"score": [88, np.nan, 92, 76, 76, 65],
"submission_date": ["2026-03-01", "03/02/2026", "2026-03-03", "2026-03-04", "2026-03-04", "2026-03-05"]
}
df = pd.DataFrame(raw_data)
def clean_academic_dataset(data: pd.DataFrame) -> pd.DataFrame:
cleaned = data.copy()
# 1. Deduplication
cleaned.drop_duplicates(inplace=True)
# 2. Text Normalization
cleaned["subject"] = cleaned["subject"].str.strip().str.title().fillna("General")
# 3. Missing Value Imputation
median_score = cleaned["score"].median()
cleaned["score"].fillna(median_score, inplace=True)
# 4. Standardize Dates
cleaned["submission_date"] = pd.to_datetime(cleaned["submission_date"], format="mixed")
return cleaned
cleaned_df = clean_academic_dataset(df)
print(cleaned_df.describe())Drop rows if missing data is greater than 30% of key target columns or cannot be reasonably estimated. Impute when sample size preservation is critical.
Our verified Ph.D. subject experts deliver 100% original, rubric-compliant assistance with Turnitin reports and step-by-step explanations.