I love discounts and I bet you are too. My fav online shopping platform is Shopee MY. But, how come I can always 24/7 checking on the App to get notified when the seller promotes the products with DISCOUNT? It is not works for me.
So, I build my own Shopee alert.
How to get the information?
The first thing is I need to get the information from the Shopee. But, do I have the Shopee API? No. I don't. My next option is web scraping. By using requests
and BeautifulSoup
, it is possible to do so with a few tricks.
...
headers = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0",
"X-Requested-With": "XMLHttpRequest",
"Referer": link,
}
page = requests.get(URL, headers=headers)
if page.status_code == 200:
soup = BeautifulSoup(page.content, "html.parser")
...
Where do I store the information?
Next, I must store the data to compare the price/discount before sending the alert. Why do we always need the alert when the price is not the lowest?
I build for my own use; I just need a simple database. Below is the database in my list:
- Google Sheet
- Baserow ✅
- Excel and store in AWS S3
After few consideration, I choose to use Baserow. But why?
- open-source with API capability
- backend using Postgres
- better admin panel
...
res = requests.get(
url=f"{base_url}database/rows/table/{table_id}/?user_field_names=true",
headers=headers,
)
if res.status_code == 200:
data = res.json()
res_data = data["results"]
df = pd.DataFrame(res_data)
...
Notification or alert
If the price is the lowest (compared with the historical data), it should alert me.
For this project, I am using Slack channel to get the alert. Slack is one of the easiest alert integration compared to Telegram or Discord.
...
def sendToSlack(itemRes, no_data=False):
if no_data:
payload = getPayloadEmpty()
else:
payload = getPayload(itemRes)
slack_webhook = os.environ.get("SLACK_URL")
requests.post(slack_webhook, json=payload)
...
Now, let's see how the alert looks like
It's contain the discount and link to the product.
Deploy the script
I am using cron job running in the AWS Lightsail instance. Schedule to run the bash
script every 3rd hour from 1 AM to 11 PM.
export PYTHONPATH=/home/amzar/Documents/python_env/shopee-tracker/bin/python3
/home/amzar/Documents/python_env/shopee-tracker/bin/python3
/home/amzar/Documents/shopee-tracker/app.py
Cron expression 0 1-23/3 * * * /home/amzar/scripts/run_shopee_tracker.sh
Full Code
Github follow the installation step in README.md
Top comments (0)