Earlier this year, I built a sample that allows you to plot your Bookmarks. However, I recently received an email asking if I knew of a service that will enable you to export your Bookmarks. So, I decided to revamp this sample to export your Bookmarks into a CSV. This example uses Python and Flask to create a website where you can export a CSV of your bookmarks.
The code is currently live using Render for hosting, and you can find the full version of the code on our GitHub.
What do you need to get started with the Twitter API
You need to sign up for access to the Twitter API. You will also create a project in the developer portal and an App containing the credentials required to use the Twitter API. Additionally, you will need to have OAuth 2.0 turned on in your App’s authentication settings. The developer playground is a great place to start using the Twitter API.
Setting up your templates
In your code editor, you will want to create three files, base.html, index.html, and next.html inside a folder called templates.
The base.html file is where you set up your site's style and HTML formatting. All the other templates you make will extend from this template.
<!DOCTYPE html>
<html lang="en">
<head>
<title>Export your Bookmarks!{% block title %}{% endblock %}</title>
<link
href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"
rel="stylesheet"
/>
<link
href="https://getbootstrap.com/examples/jumbotron-narrow/jumbotron-narrow.css"
rel="stylesheet"
/>
</head>
<body>
<div class="container">
<h1>Export your Bookmarks!</h1>
{% block containercontent %}{% endblock %}
<hr />
<div class="footer">
<p class="text-muted">
<small
>This is a basic demo created by the
<a href="https://twitter.com/TwitterDev">@TwitterDev</a> team. It is
written in Python, using
<a href="http://flask.pocoo.org/">Flask</a>. Basic CSS is provided by
<a href="http://getbootstrap.com/">Bootstrap</a>. The source code
for this project can be found at
<a href="https://github.com/twitterdev/export-bookmarks"
>github/twitterdev/export-bookmarks</a
>.</small
>
</p>
</div>
</div>
</body>
</html>
The index.html serves as the landing page for your site and walks you and your users through what you can expect from the demo site.
{% extends "base.html" %}
{% block containercontent %}
<p class="lead">This application allows you to export your recent Bookmarked Tweets.</p>
<div class="hero-unit">
<p>To get started, we'll need you to authorize our demo App (Hello world twitter bot) to access your bookmarks on Twitter using OAuth 2.0. Learn more information about <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">OAuth 2.0.</a></p>
<p>Here's how it works:</p>
<ol>
<li>We will redirect you to Twitter.com for you to authorize our app.</li>
<li>While at Twitter.com, you will authorize our App to access your Bookmarks.</li>
<li>After you click "Export CSV", you should get back a CSV of your most recent Bookmarks.</li>
</ol>
</div>
<p>Let's get started!</p>
<a href="/start" class="btn btn-small btn-success">Start</a>
{% endblock %}
The next.html template is where you can set up to download the CSV after clicking on a button. In the HTML for the button, you will find a link to next <a href=next>. This is where you will later connect to a route called next where you set the logic to download a CSV.
{% extends "base.html" %}
{% block containercontent %}
<p class="lead">This application allows you to export your recent Bookmarked Tweets.</p>
<div class="hero-unit">
<p>To get started, we'll need you to authorize our demo App (Hello world twitter bot) to access your bookmarks on Twitter using OAuth 2.0. Learn more information about <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">OAuth 2.0.</a></p>
<p>Here's how it works:</p>
<ol>
<li>We will redirect you to Twitter.com for you to authorize our app.</li>
<li>While at Twitter.com, you will authorize our App to access your Bookmarks.</li>
<li>After you click "Export CSV", you should get back a CSV of your most recent Bookmarks.</li>
</ol>
</div>
<p>Let's get started!</p>
<a href="/start" class="btn btn-small btn-success">Start</a>
{% endblock %}
Installing the required packages
In your terminal, you can install the packages you need with the following command:
pip3 requests pandas requests_oauthlib flask
We will be using requests to make HTTP requests to the Twitter API, pandas for shaping the data before you export it, requests_oauthlib to help connect to OAuth 2.0, and flask to set up a web framework for you to interact with a browser before downloading the export of your Bookmarks.
Creating your app.py file
Most of the logic for your application will take place in your app.py file. So, in your code editor, create a file called app.py.
You will first want to import all the packages you will use at the top of the file. The imports will include the packages you installed in the previous step and helper utilities such as base64, hashlib, and os, which help access environment variables and connect to OAuth 2.0.
import base64
import hashlib
import os
import re
import requests
import pandas as pd
from requests_oauthlib import OAuth2Session
from flask import (
Flask,
request,
redirect,
session,
render_template,
make_response,
)
As you do at the beginning of most Flask applications, you will want to set a variable called app and connect a random secret key.
app = Flask(__name__)
app.secret_key = os.urandom(50)
Now that you’ve set up your flask app, you will want to add some code that helps you connect to OAuth 2.0. OAuth 2.0 is an industry-standard authorization protocol that allows for greater control over an application’s scope, and authorization flows across multiple devices. Our implementation includes an Authorization Code Flow with PKCE.
With OAuth 2.0, you can set the scopes, which are your App's permissions, and these scopes will be present to the user when they log in.
You can obtain your client ID and client secret from your App’s keys and tokens inside the developer portal. The redirect URI for running the code locally should be http://127.0.0.1:5000/oauth/callback. When you deploy this to a server, you will want to add a redirect for the site you use for deployment.
client_id = os.environ.get("CLIENT_ID")
client_secret = os.environ.get("CLIENT_SECRET")
redirect_uri = os.environ.get("REDIRECT_URI")
auth_url = "https://twitter.com/i/oauth2/authorize"
token_url = "https://api.twitter.com/2/oauth2/token"
scopes = ["tweet.read", "users.read", "bookmark.read"]
code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)
code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")
Inside your terminal, you can set environment variables for CLIENT_ID, CLIENT_SECRET, and REDIRECT_URI.
export CLIENT_ID=’your_client_id’
export CLIENT_SECRET=’your_client_secret’
export REDIRECT_URI=’http://127.0.0.1:5000/oauth/callback’
Now you can create a function called get_bookmarks which makes an HTTP request to the manage Bookmarks endpoint. This request also includes a field for a date the Tweet was created, created_at, which has to be requested separately as a tweet.field to allow for more flexibility.
def get_bookmarks(user_id, token):
print("Making a request to the bookmarks endpoint")
params = {"tweet.fields": "created_at"}
return requests.request(
"GET",
"https://api.twitter.com/2/users/{}/bookmarks".format(user_id),
headers={"Authorization": "Bearer {}".format(token["access_token"])},
params=params,
)
At this point, you will want to set up the first page a user visits when they come to your site. For this site, you will display the index.html you set up earlier.
@app.route("/")
def hello():
return render_template("index.html")
After the user presses the start button, they will be taken to Twitter to log into their Twitter account and see your app's permissions which you set in the scopes.
@app.route("/start")
def demo():
global twitter
twitter = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)
authorization_url, state = twitter.authorization_url(
auth_url, code_challenge=code_challenge, code_challenge_method="S256"
)
session["oauth_state"] = state
return redirect(authorization_url)
Once the user has confirmed, you will get an access token that can be used to make requests on behalf of the user. For example, since you need the user's ID to use the manage Bookmarks endpoint, you can use authenticated user lookup endpoint to get their user ID. Afterward, you can request the manage Bookmarks endpoint to get the user’s Bookmarks and store them in a tabular fashion using a panda’s DataFrame.
@app.route("/oauth/callback", methods=["GET"])
def callback():
code = request.args.get("code")
token = twitter.fetch_token(
token_url=token_url,
client_secret=client_secret,
code_verifier=code_verifier,
code=code,
)
print(token)
user_me = requests.request(
"GET",
"https://api.twitter.com/2/users/me",
headers={"Authorization": "Bearer {}".format(token["access_token"])},
).json()
print(user_me)
user_id = user_me["data"]["id"]
bookmarks = get_bookmarks(user_id, token).json()
global df
df = pd.DataFrame(bookmarks["data"])
return render_template("next.html")
After the user authenticates, they will go to a page where they can download their Bookmarks when they click the export button.
@app.route("/oauth/next")
def export():
resp = make_response(df.to_csv())
resp.headers["Content-Disposition"] = "attachment; filename=export.csv"
resp.headers["Content-Type"] = "text/csv"
return resp
if __name__ == "__main__":
app.run(debug=True)
To run the file locally, run the following line in your terminal:
python3 app.py
Deployment
I currently have this project deployed using Render. I used a similar method described in their documentation on flask deployment. You will have to add your new REDIRECT_URI to your environment variables inside of Render and in the developer portal in your application’s Authentication Settings. Additionally, you will want to add your Client ID and Client Secret to your environment variables.
Next steps
Hopefully, this tutorial can serve as a starting point for getting started with OAuth 2.0, Flask, or web development. In addition, this code sample can be easily extended to select the fields and expansions you want from your CSV or connect to Google Sheets.
Be sure to inform us on the forums if you run into any troubles along the way, or Tweet us at @TwitterDev if this tutorial inspires you to create anything!
Top comments (0)