Introduction
While using Git I have come to learn of "Git for data", specifically Data Version Control, DVC. This is an open-source tool that works like Git to manage versioning for data science projects.
Developed by iterative to build models faster with data and experiment versioning and reproducible pipelines.
It is designed to simplify the process of tracking changes and collaborating on projects, and is increasingly becoming an essential tool for data scientists and machine learning engineers.
Why?
The main difference between Git and DVC is the purpose they both serve. Git is primarily a version control system for source code, while DVC is a version control system for data and machine learning models.
The two have a somewhat similar structure in how they are used to control versioning.
Using dvc, data experts can store and version control their datasets in a central repository, which is much like a code repository, ensuring there is seamless access to the latest project version by collaborators. This tool also allows versioning of machine learning models, which means it is easy to keep track of changes to models and experimenting with different parameters and techniques while keeping records of previous versions.
A key benefit of dvc is its seamless integration with existing machine learning frameworks like TensorFlow, PyTorch and scikit-learn, etc. Not to forget it provides a range of other useful features like data and model pipelines, automated experiments, and visualization tools. These features can be used to automate many repetitive aspects associated with data science projects.
Using for example, iterative's Studio one can automate bookkeeping tasks for example visualizing important metrics across projects, iterating faster by re-using code in a no-code environment, etc.
Example
In this example, using git we will control a Python file and also use dvc to control a data file and a trained machine learning model. We will also go step by step of how versioning works: initialize, add, commit, etc. The task instructions for both git and dvc are listed in the same block to compare the two systems.
- Initialize a repository
# Initialize git repository
git init
# Initialize dvc repository
dvc init
- Add file to created repository
# Add a file to the git repository
git add example.py
# Add data file to dvc repository
dvc add data_file.csv
- Commit changes to repository
# Commit the file to the git repository
git commit -m "Initial commit"
# Commit data file to dvc repository
git add data_file.csv.dvc
git commit -m "Add data file to dvc repository"
- Make changes
# Make changes to the file in git
echo "print('Hello, World!')" >> example.py
# Train machine learning model in dvc
python train_model.py data_file.csv
- Add changes
# Add the changes to the git repository
git add example.py
# Add trained model to dvc repository
dvc add model.pkl
- Commit changes
# Commit the changes to the repository
git commit -m "Add print statement"
# Commit trained model to dvc repository
git add model.pkl.dvc
git commit -m "Add trained model to dvc repository"
As observed above, even though both share a lot of similarities they have different commands and workflows tailored to the specific use case.
Conclusion
DVC is an essential tool for data scientists and machine learning engineers who are looking to streamline their workflow and collaborate effectively. It is a tool worth checking out for anyone doing data science/machine learning related projects.
Please note this is not meant to be a comprehensive knowledge check rather it is a quick run over what the tool is.
JokeofTheDay: Why did the data scientist use both Git and DVC?
Because he didn't want to get data-tached from his version control!
Exploring the Possibilities: Let's Collaborate on Your Next Data Venture! You can check me out at this Link
Top comments (0)