AI & Data

Data Exploration & Preparation

Module 1

Introduction to data science workflows, data formats, and preparation techniques

Data Exploration & Preparation

Common problems and solutions when dealing with data science projects

Data science requires an experimental effort
It has a lot of impacts on the model performances
Data Preparation requires knowledge about data format and their constraints
It can be a mess to organize in an optimal way

Key insight: Up to 80% of a data scientist's time is spent on data preparation!

Data Exploration & Preparation Workflow

The complete journey from raw data to a deployed model:

Data Science Pipeline

Problem: Too Many Combinations!

Obviously some combinations will be avoided
Still several transformations tries will be attempted
The execution is not linear!

Challenge: Finding the optimal combination of transformations requires systematic experimentation.

What Data Types Can Algorithms Use?

Machine learning algorithms need data in specific formats:

Numerical - integers and floats
Categorical - labels and categories

Key insight: Text, images, and complex data must be transformed into these types before ML can use them.

The DataFrame Structure

Algorithms want "rectangle" data
Data organized as a matrix
Rows = instances or observations
Columns = variables or features
One feature will be the target for predictions

Feature 1	Feature 2	Feature 3	Target
value	value	value	label
value	value	value	label
...	...	...	...

Why This Course?

Data preparation requires software engineering skills
Bridging the gap between Data Scientist and AI Engineer

From Raw Sources to ML-Ready Data

Data Exploration: Tools & Methodology

What tools and approaches should we consider?

Tool	Pros	Cons
Jupyter	Interactive, visual	Cell order issues, hard to version
Python scripts	Reproducible, versionable	Less interactive
Orange	Visual workflow, no coding	Limited customization

Data Exploration with Orange

Orange is a visual data mining and machine learning tool
Download from: orangedatamining.com
Drag-and-drop widgets for data loading, visualization, and modeling
Great for rapid prototyping and understanding data pipelines

Installation:

pip install orange3

Your Turn: Data Exploration!

Load the Titanic dataset in Orange and explore:

Load file: titanic_train.csv → How many features? rows?
Identify the target variable (Survived)
Find correlations between features and survival

Suggested Orange Workflow

Questions?

Data Exploration & Preparation

Next: Data Formats - JSON, XML, CSV

Slide Overview