Why bother
Say you get a project that doesn't have a requirements.txt file in it and that project has 20+ imports, meaning you need to install 20+ modules manually. Sound not that interesting.
That's when pipreqs comes into play as a "life saver". This tool will scan all scripts/folders in the current working directory (or where you want it to look by providing a path) and installs all the found packages.
Example usage
Solution
- create virtual env
- activate it
- install
pipreqs - tell
pipreqsto look for files in the current folder"./"and use--encoding utf-8- wait until
requirements.txtis created
- wait until
- install script dependencies from created
requirements.txt
Which results in this command:
# windows
python -m venv env && \
source env/Scripts/activate && \
pip install pipreqs && \
pipreqs --encoding utf-8 "./" && \
pip install -r requirements.txt && \
pip freeze > requirements.txt
# linux
python -m venv env && \
source env/source/activate && \
pip install pipreqs && \
pipreqs --encoding utf-8 "./" && \
pip install -r requirements.txt && \
pip freeze > requirements.txt
Low amount of imports example
Let's say you have a script like this:
import requests
response = requests.get('https://serpapi.com/playground')
print(response.html)
Big amount of imports example
The point of it is to show how all the modules install automatically without having an initial requirements.txt file.
Here will have a bigger amount of imports:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
import scipy.stats as stats
import statsmodels.api as sm
import sklearn
import yellowbrick
import wordcloud
import nltk
import spacy
import transformers
import streamlit as st
# Load and clean data
data = pd.read_csv('data.csv')
data.dropna(inplace=True)
# Descriptive statistics
print('Data Summary')
print(data.describe())
# Data visualization
sns.histplot(data['age'], kde=False, bins=10)
plt.title('Age Distribution')
plt.show()
px.scatter(data, x='income', y='age', color='gender', title='Income vs. Age')
# Correlation analysis
corr_matrix = data.corr()
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()
# Statistical analysis
stat, p = stats.ttest_ind(data[data['gender']=='M']['income'], data[data['gender']=='F']['income'])
print(f'T-test: statistic={stat}, pvalue={p}')
# Machine learning
X = data[['age', 'income']]
y = data['gender']
model = sklearn.linear_model.LogisticRegression()
model.fit(X, y)
visualizer = yellowbrick.classifier.classification_report(model, X, y)
visualizer.show()
# Text analysis
text = 'This is a sample text for text analysis'
tokens = nltk.word_tokenize(text)
print(f'Tokenized text: {tokens}')
nlp = spacy.load('en_core_web_sm')
doc = nlp(text)
for token in doc:
print(token.text, token.pos_)
model = transformers.pipeline('sentiment-analysis')
result = model(text)[0]
print(f'Sentiment analysis: {result["label"]}, score={result["score"]}')
# Streamlit app
st.title('Data Analysis App')
st.write('Data Summary')
st.write(data.describe())
Generated requirements.txt afterward:
altair==4.2.2
attrs==23.1.0
blinker==1.6.2
blis==0.7.9
cachetools==5.3.0
catalogue==2.0.8
certifi==2022.12.7
charset-normalizer==3.1.0
click==8.1.3
colorama==0.4.6
confection==0.0.4
contourpy==1.0.7
cssselect==1.2.0
cycler==0.11.0
cymem==2.0.7
decorator==5.1.1
docopt==0.6.2
entrypoints==0.4
filelock==3.12.0
fonttools==4.39.3
fsspec==2023.4.0
gitdb==4.0.10
GitPython==3.1.31
huggingface-hub==0.14.1
idna==3.4
importlib-metadata==6.6.0
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.17.3
kiwisolver==1.4.4
langcodes==3.3.0
lxml==4.9.2
markdown-it-py==2.2.0
MarkupSafe==2.1.2
matplotlib==3.7.1
mdurl==0.1.2
murmurhash==1.0.9
nltk==3.8.1
numpy==1.24.3
packaging==23.1
pandas==2.0.1
parsel==1.8.1
pathy==0.10.1
patsy==0.5.3
Pillow==9.5.0
pipreqs==0.4.13
plotly==5.14.1
preshed==3.0.8
protobuf==3.20.3
pyarrow==12.0.0
pydantic==1.10.7
pydeck==0.8.1b0
Pygments==2.15.1
Pympler==1.0.1
pyparsing==3.0.9
pyrsistent==0.19.3
python-dateutil==2.8.2
pytz==2023.3
pytz-deprecation-shim==0.1.0.post0
PyYAML==6.0
regex==2023.5.4
requests==2.29.0
rich==13.3.5
scikit-learn==1.2.2
scipy==1.10.1
seaborn==0.12.2
six==1.16.0
smart-open==6.3.0
smmap==5.0.0
spacy==3.5.2
spacy-legacy==3.0.12
spacy-loggers==1.0.4
srsly==2.4.6
statsmodels==0.13.5
streamlit==1.22.0
tenacity==8.2.2
thinc==8.1.10
threadpoolctl==3.1.0
tokenizers==0.13.3
toml==0.10.2
toolz==0.12.0
tornado==6.3.1
tqdm==4.65.0
transformers==4.28.1
typer==0.7.0
typing_extensions==4.5.0
tzdata==2023.3
tzlocal==4.3
urllib3==1.26.15
validators==0.20.0
w3lib==2.1.1
wasabi==1.1.1
watchdog==3.0.0
wordcloud==1.9.1.1
yarg==0.1.9
yellowbrick==1.5
zipp==3.15.0
Current limitations
The only drawback for now, is that it doesn't recognize all packages as there're ~10+ related issues when pipreqs didn't recognize a package.
Top comments (0)