60 mins lesson duration•8 mins read
2: NumPy Arrays & Pandas Wrangling
N-dimensional arrays, vectorized functions, DataFrame operations, grouping, and handling missing data.
NumPy and the Art of Vectorization
Underneath standard Python lists lies cell pointers, making looping slow. NumPy introduces the ndarray—a contiguous block of homogeneous memory that delegates operations to highly optimized C/Fortran libraries.
What is Vectorization?
Instead of writing explicit loops to apply an operation to every element of an array, vectorized functions perform operations on the entire array at once:
- Loop approach: $O(N)$ overhead of Python runtime type checks.
- Vectorized approach: Performed in a single CPU instruction pass.
Pandas DataFrames
Pandas extends NumPy by adding labels and indexes, creating the DataFrame. It is the ultimate tool for handling tabular data.
Essential Wrangling Techniques
- Handling Missing Values:
- Impute/fill missing entries with the column mean:
df.fillna(df.mean()) - Drop rows with excessive missing data:
df.dropna(subset=['critical_column'])
- Impute/fill missing entries with the column mean:
- Indexing & Slicing:
- Use $.loc$ for label-based indexing.
- Use $.iloc$ for integer-position indexing.
- Aggregations & GroupBy:
- Group data by categorical fields and calculate statistical aggregates (mean, sum, median).
Interactive Lesson Code Snippet
import numpy as np
import pandas as pd
# 1. NumPy Vectorized operations
arr = np.array([1, 2, 3, 4, 5])
print("Vectorized multiplication (arr * 10):", arr * 10)
# 2. Pandas Data Wrangling
data = {
'Department': ['IT', 'HR', 'IT', 'Marketing', 'HR', 'IT'],
'Salary': [85000, 60000, 95000, 70000, None, 90000],
'Experience': [3, 2, 5, 4, 1, 4]
}
df = pd.DataFrame(data)
# Impute missing salary values with the mean of the column
mean_salary = df['Salary'].mean()
df['Salary'] = df['Salary'].fillna(mean_salary)
# Perform Groupby aggregation
dept_summary = df.groupby('Department').agg(
Average_Salary=('Salary', 'mean'),
Total_Employees=('Salary', 'count'),
Avg_Experience=('Experience', 'mean')
).round(2)
print("
Original DataFrame with Imputed Salary:")
print(df)
print("
Department Summary Analysis:")
print(dept_summary)Language: python
Lesson Code (Python)
import numpy as np
import pandas as pd
# 1. NumPy Vectorized operations
arr = np.array([1, 2, 3, 4, 5])
print("Vectorized multiplication (arr * 10):", arr * 10)
# 2. Pandas Data Wrangling
data = {
'Department': ['IT', 'HR', 'IT', 'Marketing', 'HR', 'IT'],
'Salary': [85000, 60000, 95000, 70000, None, 90000],
'Experience': [3, 2, 5, 4, 1, 4]
}
df = pd.DataFrame(data)
# Impute missing salary values with the mean of the column
mean_salary = df['Salary'].mean()
df['Salary'] = df['Salary'].fillna(mean_salary)
# Perform Groupby aggregation
dept_summary = df.groupby('Department').agg(
Average_Salary=('Salary', 'mean'),
Total_Employees=('Salary', 'count'),
Avg_Experience=('Experience', 'mean')
).round(2)
print("
Original DataFrame with Imputed Salary:")
print(df)
print("
Department Summary Analysis:")
print(dept_summary)Console Output
Vectorized multiplication (arr * 10): [10 20 30 40 50]
Original DataFrame with Imputed Salary:
Department Salary Experience
0 IT 85000.0 3
1 HR 60000.0 2
2 IT 95000.0 5
3 Marketing 70000.0 4
4 HR 80000.0 1
5 IT 90000.0 4
Department Summary Analysis:
Average_Salary Total_Employees Avg_Experience
Department
HR 70000.0 2 1.5
IT 90000.0 3 4.0
Marketing 70000.0 1 4.0Test Your Knowledge
Instant feedbackQuick Check: NumPy & Pandas
How do you fill missing values with a column's mean?
Up next · Continue learning
Visualizing Patterns with Seaborn & Matplotlib
Building distribution charts, relational scatter plots, and correlation heatmaps.
6 mins read50 mins
Start next lesson