Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing.
args/*args review
Check out this post for a review of how *args
**kwargs
work in python.
python args and kwargs article by @_waylonwalker
All Kedro inputs are catalog Entries
When kedro runs your pipeline it uses the catalog to imperatively load your data, meaning that you don't tell kedro how to load your data, you tell it where your data is and what type it is. These catalog entries are like a key-value
store. You just need to give the key when setting up a node.
Single Inputs
These are fairly straightforward to understand. In the example below when kedro
runs the pipeline it will load the input from the catalog, then pass that input to the func, then save the returned value to the output catalog entry.
from kedro.pipeline import node
def create_int_sales(sales):
"cleans up raw sales data"
...
return cleaned_sales
my_node = node(
func=create_int_sales,
inputs='raw_sales',
output='int_sales',
)
List of inputs
Let's look at an example node that combines more than one dataset. When kedro has sees a list of catalog entries it will load up each catalog entry sequentially then pass them in order to the create_sales_report
function.
from kedro.pipeline import node
def create_sales_report(sales, products):
"adds product metadata to the sales data"
...
my_node = node(
func=create_sales_report,
inputs=['pri_sales', 'pri_products'],
output='sales_report',
)
simulating pipeline run using 2 inputs
Here We can simulate what kedro does during the pipeline run by using *args
.
# inputs you gave kedro
inputs=['pri_sales', 'pri_products']
# load data
input_data = [catalog.load(entry) for entry in inputs]
# run the node
sales_report = create_sales_report(*input_data)
# save the data to the output
catalog.datasets.sales_report.save(sales_report)
More generalizable functions
We can also use *args
to make our functions a little bit more generalizable. The first that
comes to my mind is a unioner. The second
def unioner(*dfs: pd.DataFrame): -> pd.DataFrame
return pd.concat(dfs)
Now we can pass any number of DataFrames into our kedro node to get unioned together, but
do we really need a function for a one-liner... No we can use an inline function for this case.
my_node = node(
func=unioner,
input=['sales_2017', 'sales_2018'],
output='sales',
)
*args
scares the crap out of me!
It's great for the unioner
example where its a collection of similar things where order
does not matter. But for the create_sales_report
function. Those are distinctly different
inputs. If someone does some refactoring and changes the order in one place or another it's
going to turn into a bad day real fast.
**kwargs are a bit better
Let's refactor the create_sales_report
before someone tries to ruin our day. We can easily
do this by passing a dictionary (keys are the argument name, values are the catalog key)
of arguments to kedro instead of a list.
from kedro.pipeline import node
def create_sales_report(sales, products):
"adds product metadata to the sales data"
...
my_node = node(
func=create_sales_report,
inputs={'sales': 'pri_sales', 'products': 'pri_products'},
output='sales_report',
)
Now if someone tries to refactor the order of arguments we are safe!
Simulating the pipeline run with **kwargs
Pretty much the same as before, except with **kwargs
and dictionaries
keeping us a bit
safer.
# inputs you gave kedro
inputs={'sales': 'pri_sales', 'products': 'pri_products'},
# load data
input_data = {arg: catalog.load(entry) for arg, entry in inputs.items()}
# run the node
sales_report = create_sales_report(**input_data)
# save the data to the output
catalog.datasets.sales_report.save(sales_report)
Stay Safe
Kedro inputs are quite easy to understand if you already have a grasp of *args
and **kwargs
and if you don't it is still fairly intuitive to pick up. Stay on the safe side, if your
collection of inputs are clearly different things, use a dictionary for safety.
π see an issue, edit this post on GitHub
I have been writing short snippets about my mentality breaking into the tech/data industry in my newsletter, π check it out, and let's get the conversation started.
Top comments (2)
I'm definitely pointing some of the teams that struggle with this problem to your post!
Thanks @yetudada ! For sure, I think this is one that we take for granted once we grasp the concept and forget that it's not intuitive the first time you see it!