Overview of My Submission
I loaded over 300,000 Public Venmo Transactions into a Mongo Atlas database and use Atlas' Full Text Search to search by transaction memo.
Venmo transactions are interesting for full text search because users have to leave a public memo on every transaction. What relationships can you infer between Venmo users, based on their transaction memo? Are they friends, roommates, family, coworkers, lovers, or something else?
Deployed Application:
https://reidjs.github.io/venmo-search-frontend/
Submission Category:
Choose Your Own Adventure
Link to Code
Frontend Repo:
Venmo Search
Deployed here:
https://reidjs.github.io/venmo-search-frontend/
About
Uses MongoDB's Atlas Search to search hundreds of thousands of Venmo transactions. Made for dev.to's mongodb hackathon
Why
I built this to explore using MongoDB's Atlas Search as an alternative to ElasticSearch for full text search. I used Venmo data because
- it's public
- users are forced to leave a text "comment" for every transaction
Usage
Production
- Go to https://reidjs.github.io/venmo-search-frontend/
- Type a query
Local Development
- Start backend https://github.com/reidjs/venmo-search-server
- Start frontend with
npm run dev
- Open browser to localhost:8000
Notes
Venmo dataset from https://github.com/sa7mon/venmo-data
- Out of the 7,000,000 transactions, I only used about 342,000 venmo transactions (free tier Mongo Atlas)
- To decide which transactions went into the database, I 'ranked' each transaction according to how long the comment was. Longer comment = more likely to be in database.
How to load in BSON data to atlas:
mongorestore --uri mongodb+srv://[USERNAME]:[PASSWORD]@cluster0.abbj1.mongodb.net/[DATABASE] venmo.bson
How to Load in JSON dataβ¦
Additional Resources / Info
Backend Repo:
https://dashboard.heroku.com/apps/venmo-search
https://venmo-search.herokuapp.com/?q=hello
deploy changes to heroku:
npm run deploy
Run locally:
npm start
Looped through 7 million records to calculate the score on each one This nearly set my 2.6GHZ 6Core i7 32GB MBPro on fire In hindsight maybe I should have calculated a virtual field w/ mongoose, but this was the easy, dumb way of doing it. After about an hour I had at least 500,000 scores calculated, which was a big enough dataset to load into atlas
Top comments (3)
it is not working
Thanks for letting me know! Should be fixed now.
Yep