DEV Community

Implement real-time updates with Django REST Framework | Building Cryptocurrency API

Rashid on December 15, 2019

This post cross-published with OnePublish What's up DEVs? It is time to create another cool project using Django and REST Framework. In this pos...

Read full post

ChristianGeng • Dec 17 '19

Lovely, thanks for sharing this!
When trying to get it to work, I have a potentially dumb error:
I am getting the Rest Entry Point at localhost:8000, however the list of data stays empty.
Same with the default database db.sqlite3 where no data are arriving either (the table is created after doing the migrations, but the select * returns nothing).
However, the celery task works in that it prints the data updates to the console (starting with
Updating data... and the new bitcoin etc. rates).
Now I wonder: am I making a kind of typical, very dumb mistake?

Rashid • Dec 17 '19

Did you migrate your project? Try to delete all migrations and migrate your project right after you created models. You can check my github I put commands there. Let me know if it worked for you :)

ChristianGeng • Dec 17 '19

Thank you very much for the fast response! I have now rebuilt everything (also a fresh install of a venv with the requirements), but I am still getting the same problem:

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc"  -delete

rm -v db.sqlite3
python3 manage.py makemigrations trackingAPI
python3 manage.py migrate
python3 manage.py runserver
celery -A cryptocurrencytracking worker -l info

I also tried a completely fresh checkout with a git clone, but again the same issue (Database gets created, but no data end up there. Rest is visible on localhost:8000, but no data arrive, only the celery worker prints the data as intendended)

Rashid • Dec 17 '19

Hmm, I will check it again in few hours. Can you please update your tasks.py file:

# tasks.py

from time import sleep
from celery import shared_task
from bs4 import BeautifulSoup
from urllib.request import urlopen, Request

from .models import Cryptocurrency


@shared_task
# do some heavy stuff
def crawl_currency():
    print('Crawling data and creating objects in database ..')
    req = Request('https://coinranking.com', headers={'User-Agent': 'Mozilla/5.0'})
    html = urlopen(req).read()
    bs = BeautifulSoup(html, 'html.parser')
    # Find first 5 table rows
    rows = bs.find('tbody', class_="table__body").find_all('tr', class_="table__row")[0:5]
    for row in rows:
        cryptocurrency = row.find('span', class_="profile__name").get_text().strip().replace('\n', '')
        values = row.find_all('div', class_="valuta")
        price = values[0].get_text().strip().replace('\n', '')
        market_cap = values[1].get_text().strip().replace('\n', '')
        change = row.find('div', class_="change").find('span').get_text().strip().replace('\n', '')
        print({'cryptocurrency': cryptocurrency, 'price':price, 'market_cap':market_cap, 'change':change}) 
        # Create object in database from crawled data 
        Cryptocurrency.objects.create(
            cryptocurrency = cryptocurrency,
            price = price,
            market_cap = market_cap,
            change = change
        )
        # Sleep 3 seconds to avoid any errors
        sleep(3)

crawl_currency()

and go check your admin to see objects are created.

Make sure you are running celery and django at the same time.

ChristianGeng • Dec 17 '19

Thanks a lot, that was it!

not Cryptocurrency.objects

is always false, so crawl_currency() was never called.
I have set a celery breakpoint there for the first time, then called crawl_currency() at the breakpoint by hand, and now it works like a charm!

Thanks so much!

P.S.:
Would it make sense to test for an empty queryset instead? Sth. like this:

not Cryptocurrency.objects.all()

Rashid • Dec 17 '19

Great! Ah, I see. Yeah I will fix that line. Thank you for your attention👍

Marcelo C. Lopes • Oct 22 '20

Hello guys, because when this code snippet is active, rabbitmq is idle.

if not Cryptocurrency.objects:

crawl_currency()

while True:
sleep(15)
update_currency()

I didn't understand this logic, could someone help me

Hamza • Feb 25 '21

thank you sir for such an amazing article. Now I am working on a similar project but instead of refreshing I want my API to automatically stream data whenever new data comes