DEV Community

Cover image for Venmo Search
reid j sherman
reid j sherman

Posted on

Venmo Search

Overview of My Submission

I loaded over 300,000 Public Venmo Transactions into a Mongo Atlas database and use Atlas' Full Text Search to search by transaction memo.
Venmo transactions are interesting for full text search because users have to leave a public memo on every transaction. What relationships can you infer between Venmo users, based on their transaction memo? Are they friends, roommates, family, coworkers, lovers, or something else?

Deployed Application:

https://reidjs.github.io/venmo-search-frontend/

Submission Category:

Choose Your Own Adventure

Link to Code

Frontend Repo:

Venmo Search

Deployed here:

https://reidjs.github.io/venmo-search-frontend/

About

Uses MongoDB's Atlas Search to search hundreds of thousands of Venmo transactions. Made for dev.to's mongodb hackathon

demo

Why

I built this to explore using MongoDB's Atlas Search as an alternative to ElasticSearch for full text search. I used Venmo data because

  1. it's public
  2. users are forced to leave a text "comment" for every transaction

Usage

Production

  1. Go to https://reidjs.github.io/venmo-search-frontend/
  2. Type a query

Local Development

  1. Start backend https://github.com/reidjs/venmo-search-server
  2. Start frontend with npm run dev
  3. Open browser to localhost:8000

Notes

Venmo dataset from https://github.com/sa7mon/venmo-data

  • Out of the 7,000,000 transactions, I only used about 342,000 venmo transactions (free tier Mongo Atlas)
  • To decide which transactions went into the database, I 'ranked' each transaction according to how long the comment was. Longer comment = more likely to be in database.

How to load in BSON data to atlas: mongorestore --uri mongodb+srv://[USERNAME]:[PASSWORD]@cluster0.abbj1.mongodb.net/[DATABASE] venmo.bson

How to Load in JSON data…

Additional Resources / Info

Backend Repo:

https://dashboard.heroku.com/apps/venmo-search

https://venmo-search.herokuapp.com/?q=hello

deploy changes to heroku: npm run deploy

Run locally: npm start

Looped through 7 million records to calculate the score on each one This nearly set my 2.6GHZ 6Core i7 32GB MBPro on fire In hindsight maybe I should have calculated a virtual field w/ mongoose, but this was the easy, dumb way of doing it. After about an hour I had at least 500,000 scores calculated, which was a big enough dataset to load into atlas




Deployed application

Top comments (3)

Collapse
 
codewithpom profile image
Padmashree Jha

it is not working

Collapse
 
reidjs profile image
reid j sherman

Thanks for letting me know! Should be fixed now.

Collapse
 
codewithpom profile image
Padmashree Jha

Yep