pandas foundations: loading, selecting and validating data

What you will learn

Read a CSV, inspect types, select rows and repair a controlled dirty copy without silently changing the original.

Before you start

You need basic variables, lists and function calls, not advanced Python algorithms. If these are new, visit Python foundations first.

1. Create a local environment: python -m venv .venv. Activate it for your OS and install python -m pip install jupyterlab numpy pandas matplotlib seaborn plotly.
2. Download the 12-row sales CSV into your notebook folder. Keep its filename unchanged.
3. Start Jupyter with jupyter lab. Run the setup below in a fresh notebook before every workshop.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
raw = pd.read_csv('data-science-sales.csv', parse_dates=['date'])
df = raw.copy()
df['revenue'] = df['units'] * df['unit_price']
net = df.loc[df['returned'].eq(0)].copy()

Data dictionary: each row is one order for one product; units is a positive count; unit_price is INR per unit; returned is 0 or 1. Revenue is units × unit_price. For this workshop, net sales exclude the entire returned order (not partial returns). The CSV has 12 rows, 4950 gross revenue, 4450 net revenue and 35 net units. This small fictional sample teaches technique; it does not justify population or causal claims.

Save your notebook and written interpretation. There is no browser code judge and no automatic grading of visual design.

Official pandas documentation

Learn the idea

A DataFrame has labelled columns and an index. Use head(), info() and shape before analysis. loc selects labels and masks; iloc selects positions. Missing values are not zeros: choose a repair only when the business rule permits it. Use a new DataFrame for cleaning experiments. Drop duplicates on the business key, not every repeated value. Avoid chained assignment; use .loc or .assign and retain a raw copy.

Follow a worked example

print(raw.shape)
print(raw['order_id'].is_unique)
print(raw.isna().sum().sum())

Expected result

(12, 7), True, 0. The source CSV is clean; dirty-data exercises deliberately create a copy.

Skill-development workshop

These are practical assignments, not MCQs or code-judge puzzles. Work in your own notebook or MySQL client. Complete the levels in order; if a task feels difficult, follow the hint and compare with the reference only after making an attempt. Keep your result and a short explanation of what it means.

Easy: Select order_id and product for the first three rows.

Success criteria: Three rows: Notebook, Pen, Lamp; two columns.

print(raw[['order_id','product']].head(3))

Medium: Filter South orders with at least two units.

Success criteria: Order ids [2,5,11].

result=raw.loc[(raw['region']=='South') & (raw['units']>=2)]
assert result['order_id'].tolist()==[2,5,11]

Hard: Repair a duplicated row and a missing region in a dirty copy.

Success criteria: 12 unique orders, no missing region; the deliberately missing value is labelled Unknown, not guessed.

dirty=pd.concat([raw,raw.iloc[[0]]],ignore_index=True)
dirty.loc[0,'region']=None
clean=dirty.assign(region=dirty['region'].fillna('Unknown')).drop_duplicates('order_id',keep='first')
assert len(clean)==12 and clean['order_id'].is_unique
assert clean['region'].isna().sum()==0

Professional: Write a data-quality gate before revenue reporting.

Success criteria: Reject duplicate ids, nonpositive units/prices, invalid returns, missing dates. All checks pass on raw.

assert raw['order_id'].is_unique
assert raw[['order_id','date','region','product','units','unit_price','returned']].notna().all().all()
assert raw['units'].gt(0).all() and raw['unit_price'].gt(0).all()
assert raw['returned'].isin([0,1]).all()
assert pd.api.types.is_datetime64_any_dtype(raw['date'])

Reflection and submission

Save your queries/notebook and outputs. Write three sentences: what you changed, how you checked it, and one limitation. A correct chart or query without a clear interpretation is not a completed professional assignment.