Victory Akaniru

Posted on Feb 23, 2020

The one where we build a web scrapper and a slackbot - Part 2

#slackbot #apscheduler #slackapi #python

The problem

In part one of this two-part series we set out on a quest to automate these tasks

Visit Brainyquote
Find and copy a random quote about excellence from the site.
Post the quote to a slack channel.

we were able to build a web scraper that enabled us to scrape a website for quotes and save them to a JSON file. In this second part of the article series, we would be bringing it all together by building a slackbot that would periodically post one of the quotes we scrapped to a slack channel!

If you missed part one you can find it here

The Product

Slackbot: Here we would visit slack apps page, and create a bot associated with a slack channel.
Slackapi backend: Making use of slacks open API we would write some python functions that when called posts the message to a channel.
Apscheduler: Finally we would write another function that would help us call those API functions at intervals.

Prerequisites

Building up from part one, all you'll need at this point is a slack workspace, you could easily create one for free here or just build the bot into any workspace by creating a channel where you can play around with.

Part 2: The slackbot

To get started visit slack a modal should pop up like this

Type in your app name
Select a development workspace(slack workspace where the bot would be installed)
Click create App and you'll be navigated to this page
Now we need to set a scope for our bot

Scopes give your bot permission to do things like sending a message in your development workspace.
Navigate to OAuth & Permissions on the sidebar to add scopes to your app
Scroll down to the Bot Token Scopes section and click Add an OAuth Scope.
Select or search for chat:write scope to grant your bot the permission to post messages in channels it's a member of. you should see a success banner after this ✅
Next, we have to install the bot to our workspace, by scrolling to the top of the OAuth & Permissions pages and clicking the green Install App to Workspace button.
After the above step, you should be navigated to the below page. All you have to do is click the allow button.
Finally 🎉, copy your bot token, you'll need it to communicate with slack API.

Next steps... We need to write our backend code to interface with slack. we will be doing this in the same folder where we created our web scrapper you can find that code if you don't have it by clicking here



- cd clonedOrExistingProject
- mkdir bot
- cd scrapping-tool
- touch __init__.py builder.py post_message.py

Create and activate a virtual environment if you don't have one already



- virtualenv --python=python3 venv
- source venv/bin/activate

In the builder.py file add the following code



def construct_payload(HEADER_BLOCK, DIVIDER_BLOCK, MESSAGE_BLOCK):

    return {
        "channel": "your-channel-id-goes-here",
        "username": "name-of-your-app",
        "icon_emoji": ":robot_face:",
        "text": ":wave: Your daily dose of motivation has arrived",
        "blocks": [
            HEADER_BLOCK,
            DIVIDER_BLOCK,
            MESSAGE_BLOCK
        ],
    }

Slack has a tool called the block builder kit, it's basically used to generate markdowns that can be used to style the appearance of messages on slack. Try it out here.

For us, we created a function that accepts various blocks from the block builder kit and in the end, we want to have something looking like this

From the code, we passed in

Channel: The id of a Slack channel where we want to invite our bot(you can copy it from your slack channel, make sure to invite the bot to this channel)
Username: which would be the name of the app you created earlier
Text: would be what appears on the slack notification popup.
Blocks: contains the various parts of the message, the header, a divider line, and the actual message.

Let's start by creating our random quote generator function, this file will read our previously scrapped json quotes and return any random one when the function is called. from the root of your project folder, run the command



- mkdir utils
- touch __init__.py random_quote.py

In the random_quote file, add the following code



import random
import json


def random_quotes_generator():
    with open('quotes.json', 'r') as f:
        quotes_dict = json.load(f)


        quote = random.choice(quotes_dict)
        return quote

This file creates a function random_quotes_generator reads our quotes from the json file and with python's random module returns one random quote when called.

Next, we create our post message function. Install these packages



pip3 install slack dotenv

slack to interface with slack API and dotenv to load in our bot secret token.

Navigate to the post_message.py file and post the following code.



import os
from slack import WebClient
from dotenv import load_dotenv
from bot.builder import construct_payload
from utils.random_quote  import random_quotes_generator

load_dotenv()

SLACK_BOT_TOKEN = os.getenv("SLACK_BOT_TOKEN")

# Initialize a Web API client
slack_web_client = WebClient(token=SLACK_BOT_TOKEN)


def post_quote_to_channel():
    quote = random_quotes_generator()
    author = quote['author']
    link = f"https://www.brainyquote.com{quote['author-link']}"


    # Get the onboarding message payload
    message = construct_payload({
        "type": "section",
        "text": {
                "type": "mrkdwn",
                "text": "*Your daily dose of motivation* :blush:"
        }
    },
        {
        "type": "divider"
    },
        {
        "type": "section",
        "text": {
            "type": "mrkdwn",
            "text": f"The test of the artist does not lie in the will with which he goes to work, but in the excellence of the work he produces... *By <{link}|{author}>*"
        }
    })
    # Post the onboarding message in Slack
    response = slack_web_client.chat_postMessage(**message)

So what are we doing here...

We imported a couple of 3rd party packages, our message builder function and our random_quotes_generator.

We also load in our SLACK_BOT_TOKEN, create a .env file use the key SLACK_BOT_TOKEN and set it to your bot token.



SLACK_BOT_TOKEN=token

We initialized slack WebClient and created the post_quote_to_channel inside which we call our random generator function and save it to a variable quote.
We construct a link to each author's page.
Next, we construct our message by calling the construct_payload function exported from the message builder file and passing in all required parameters.
Finally, we call a function chat_postMessage from slack WebClient and pass in our constructed message.

You can test that this works now by calling the function on your terminal running this command



python3 post_message.py

The final step in this process would be to create our scheduler to periodically call the post_quote_to_channel. to do this we need to install some packages, create a flask server and our scheduler function.

On the root of your project folder create a new file app.py and run the following commands



pip3 install APScheduler Flask



- cd bot
- mkdir jobs
- touch __init__.py scheduler.py

In bot/jobs folder locate the scheduler.py file and paste in the following code



from apscheduler.schedulers.background import BackgroundScheduler

# jobs
from bot.post_message import post_quote_to_channel

scheduler = BackgroundScheduler()

scheduler.add_job(
    post_quote_to_channel,
    'interval',
    seconds=10
)

Here we import BackgroundScheduler from the scheduler package and initialize it.

We also import the post_quote_to_channel function from the file post_message in the bot folder.
With that, we call add_job function from the scheduler, pass the post_quote_to_channel function, the string interval and time interval in seconds. to read more about app scheduler click here

with that, we have our scheduler function ready.

Bringing it all together

In the app.py file we created earlier, post the following code



import os
import time
from flask import Flask
from bot.jobs.scheduler import scheduler

# Initialize a Flask app to host the events adapter
app = Flask(__name__)

if __name__ == "__main__":
    scheduler.start()
    print('Press Ctrl+{0} to exit'.format('Break' if os.name == 'nt' else 'C'))

    try:
        # This is here to simulate application activity (which keeps the main thread alive).
        while True:
            time.sleep(5)
    except (KeyboardInterrupt, SystemExit):
        # Not strictly necessary if daemonic mode is enabled but should be done if possible
        scheduler.shutdown()

Here we import our scheduler, create a flask server and call our scheduler function. we also use a nifty trick from StackOverflow to keep our main thread alive.

Run the program