You can acecces the full pipeline here
Mage
In my last post, i writed about a dashboard I builted using Python and Looker Studio, to visualize my powrlifting training data. In this post I will walk you through the step by step of a ETL (Extract, Transform, Load) pipeline, using the same dataset.
To build the pipeline we will use Mage to orchestrate the pipeline and Python for transforming and loading the data, as a last step we will export the transformed data into a DuckDB database.
To execute Mage, we're a gointo use the oficial docker image:
docker pull mageai/mageai:latest
The pipeline will look like this:
Extract
The extraction will be simple, we just have to read a csv file and create a pandas dataframe with it, so we can procide to the next steps. Using the Data loader block, we already have a tamplate to work with, just remember to set the "sep" parameter int he read_csv( ) function, só the data is loaded correct.
from mage_ai.io.file import FileIO
import pandas as pd
if 'data_loader' not in globals():
from mage_ai.data_preparation.decorators import data_loader
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
@data_loader
def load_data_from_file(*args, **kwargs):
filepath = 'default_repo/data_strong.csv'
df = pd.read_csv(filepath, sep=';')
return df
@test
def test_output(output, *args) -> None:
assert output is not None, 'The output is undefined'
Transform
In this step, using the Transformer block, that have a lot of templates to chose from, we will select a custom template.
The transformation we have to do is basicly the maping of the Exercise Name column, so we can identify which body part correspond to the specific exercise.
import pandas as pd
if 'transformer' not in globals():
from mage_ai.data_preparation.decorators import transformer
if 'test' not in globals():
from mage_ai.data_preparation.decorators import test
body_part = {'Squat (Barbell)': 'Pernas',
'Bench Press (Barbell)': 'Peitoral',
'Deadlift (Barbell)': 'Costas',
'Triceps Pushdown (Cable - Straight Bar)': 'Bracos',
'Bent Over Row (Barbell)': 'Costas',
'Leg Press': 'Pernas',
'Overhead Press (Barbell)': 'Ombros',
'Romanian Deadlift (Barbell)': 'Costas',
'Lat Pulldown (Machine)': 'Costas',
'Bench Press (Dumbbell)': 'Peitoral',
'Skullcrusher (Dumbbell)': 'Bracos',
'Lying Leg Curl (Machine)': 'Pernas',
'Hammer Curl (Dumbbell)': 'Bracos',
'Overhead Press (Dumbbell)': 'Ombros',
'Lateral Raise (Dumbbell)': 'Ombros',
'Chest Press (Machine)': 'Peitoral',
'Incline Bench Press (Barbell)': 'Peitoral',
'Hip Thrust (Barbell)': 'Pernas',
'Agachamento Pausado ': 'Pernas',
'Larsen Press': 'Peitoral',
'Triceps Dip': 'Bracos',
'Farmers March ': 'Abdomen',
'Lat Pulldown (Cable)': 'Costas',
'Face Pull (Cable)': 'Ombros',
'Stiff Leg Deadlift (Barbell)': 'Pernas',
'Bulgarian Split Squat': 'Pernas',
'Front Squat (Barbell)': 'Pernas',
'Incline Bench Press (Dumbbell)': 'Peitoral',
'Reverse Fly (Dumbbell)': 'Ombros',
'Push Press': 'Ombros',
'Good Morning (Barbell)': 'Costas',
'Leg Extension (Machine)': 'Pernas',
'Standing Calf Raise (Smith Machine)': 'Pernas',
'Skullcrusher (Barbell)': 'Bracos',
'Strict Military Press (Barbell)': 'Ombros',
'Seated Leg Curl (Machine)': 'Pernas',
'Bench Press - Close Grip (Barbell)': 'Peitoral',
'Hip Adductor (Machine)': 'Pernas',
'Deficit Deadlift (Barbell)': 'Pernas',
'Sumo Deadlift (Barbell)': 'Costas',
'Box Squat (Barbell)': 'Pernas',
'Seated Row (Cable)': 'Costas',
'Bicep Curl (Dumbbell)': 'Bracos',
'Spotto Press': 'Peitoral',
'Incline Chest Fly (Dumbbell)': 'Peitoral',
'Incline Row (Dumbbell)': 'Costas'}
@transformer
def transform(data, *args, **kwargs):
strong_data = data[['Date', 'Workout Name', 'Exercise Name', 'Weight', 'Reps', 'Workout Duration']]
strong_data['Body part'] = strong_data['Exercise Name'].map(body_part)
return strong_data
@test
def test_output(output, *args) -> None:
assert output is not None, 'The output is undefined'
An intresting feature of Mage is that we can visualize the changes we are making inside the Tranformer block, using the Add chart opting it's possible to generate a pie graph using the column Body part.
Load
Now is time to load the data to DuckDB. In the docker image we already have DuckDB, so we just have to include another block in our pipeline. Lets inlcude the Data Exporter block, with a SQL template so we can create a table and insert the data.
CREATE OR REPLACE TABLE powerlifting
(
_date DATE,
workout_name STRING,
exercise_name STRING,
weight STRING,
reps STRING,
workout_duration STRING,
body_part STRING
);
INSERT INTO powerlifting SELECT * FROM {{ df_1 }};
Conclusion
Mage is a powrfull tool to orchestrate pipelines, provide a complete set of templates for developing especific tasks involving ETL. In this post we had a breathh explanation about how to get start using Mage to build pipelines of data. In future post we're gointo explore more about this amizing framework.
Top comments (0)