Joblib is a Python library designed to facilitate efficient computation and useful for tasks involving large data and intensive computation.
Joblib tools :
Serialization: Efficiently saving and loading Python objects to and from disk. This includes support for
numpy arrays,scipy sparse matrices, andcustom objects.Parallel Computing: Parallelizing tasks to utilize multiple CPU cores, which can significantly speed up computations.
Using Python for Parallel Computing
Threading: The threading module allows for the creation of threads. However, due to the GIL, threading is not ideal for CPU-bound tasks but can be useful for I/O-bound tasks.
Multiprocessing: The multiprocessing module bypasses the GIL by using separate memory space for each process. It is suitable for CPU-bound tasks.
Asynchronous Programming: The asyncio module and async libraries enable concurrent code execution using an event loop, which is ideal for I/O-bound tasks.
managing parallelism manually can be complex and error-prone. This is where joblib excels by simplifying parallel execution.
Using Joblib to Speed Up Your Python Pipelines
- Efficient Serialization
from joblib import dump, load
# Saving an object to a file
dump(obj, 'filename.joblib')
# Loading an object from a file
obj = load('filename.joblib')
- Parallel Computing
from joblib import Parallel, delayed
def square_number(x):
"""Function to square a number."""
return x ** 2
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Parallel processing with Joblib
results = Parallel(n_jobs=-1)(delayed(square_number)(num) for num in numbers)
print("Input numbers:", numbers)
print("Squared results:", results)
output
Input numbers: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Squared results: [1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
- Pipeline Integration
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import joblib
# Load example dataset (Iris dataset)
iris = load_iris()
X = iris.data
y = iris.target
# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Create a pipeline
pipeline = Pipeline([
('scaler', StandardScaler()),
('svc', SVC())
])
# Train the pipeline
pipeline.fit(X_train, y_train)
# Save the pipeline
joblib.dump(pipeline, 'pipeline.joblib')
# Load the pipeline
pipeline = joblib.load('pipeline.joblib')
# Use the loaded pipeline to make predictions
y_pred = pipeline.predict(X_test)
# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy}")
output
Accuracy: 1.0
Top comments (0)