System Design Sketches

Functional requirements:
------------------------
- Follow people
- See followers and falowees
- Create posts (text, image, url, @mentions)
- Like posts
- View user feed
- View home feed


Non-functional requirements:
----------------------------
- Add post <500ms
- View feed <500ms
- Post in feed with ~2 mins delay is ok
- Billions of users
- Unlimited followers


Core entities:
--------------
- User
- Follow
- Post


API (RPC-like):
---------------
- POST /getFollowers     {userId}
- POST /getFollowees     {userId}
- POST /followUser       {userId}

- POST /createPost       {text}
- POST /likePost         {postId}

- POST /getHomeFeed      {userId}
- POST /getUserFeed      {userId}

Functional requirements:
------------------------
- Rider gets estimated fare (by Start and Destination locations)
- Rider requests a ride based on the estimate
- Driver accepts/denies request + Navigates to pickup/drop-off


Non-functional requirements:
----------------------------
- Low latency for rider-driver matching (<1min)
- Consistency of rider-driver 1:1 matching
  - For one ride, only one driver will get request at a time
- Highly available outside matching
- Handle high-throughput (peak hours, popular events)


Core entities:
--------------
- Rider
- Driver
- Ride
- Location


API:
---------------
- POST /getEstimate({from, to}) --> Partial<Ride>

- POST /requestRide({ride_id}) --> Ride

- POST /getRide({ride_id})

  - gets the latest ride entity to display the
    ride details for both rider and driver apps

- POST /updateLocation({lat, lon})

  - both driver and rider can update their locations
    which may be used to display their on the map
  - drivers might update their location i.e. once
    per 5 seconds (it is needed for matching)
  - however this can be optimised on the client side,
    where we might say that location update is
    proportional to driver speed: for still drivers
    we update location rarely, for moving - more often

- POST /acceptRide({ride_id, accept})
  - ride is being accepted by the driver

- POST /updateRide({ride_id, status})

Functional requirements:
------------------------
- Group chat (<100 users)
- Send message
- Receive message
- Persist messages (user messages are
  available on multiple devices)


Non-functional requirements:
----------------------------
- Low latency for message delivery (<500ms)
- No lost messages (delivery guarantee)


Core entities:
--------------
- Users
- Groups
- Messages


API:
---------------
- createGroup({users: [], name: ''}) -> {group_id}
- joinGroup({group_id}) -> "ACK"/ "ERR"
- quitGroup({group_id}) -> "ACK"/ "ERR"
- getMyGroups() -> {groups: []}
- getGroupMessages({group_id, max_message_id, min_message_id}) -> {messages: []}

WebSocket:
- connect()
- sendMessage({group_id, message, media: []}) -> "ACK"/ "ERR"
- onGroupUpdated({group_id, participants}) -> "ACK"
- onMessage({chat_id, user_id, message, attachments}) -> "ACK"

Functional requirements:
------------------------
- Upload video
- Watch video (read-heavy)
- Comment on video


Non-functional requirements:
----------------------------
- Reliable uploads (no corrupted/missing videos)
- Read-heavy (read/write ratio ~100:1)
- Availability > Consistency (1-2 min delay between upload and feed is ok)
- Low latency (before video starts playing)


Core entities:
--------------
- User
- Video Metadata
- Video Chunks
- Users
- Comments


API (RPC-like):
---------------
- POST /initiateUpload  {title, description, video, ...}}
  - server responds with S3 private upload URL
  - the actual video upload is going directly from client to S3

- GET /watch   {video_id}
  - returns video chunks metadata
    - S3 (or CDN) URLs to manifest/playlist files
  - video streaming happens directly from CDNs
  - download by chunks (buffering, ~1Mb), no need to load all

Functional requirements:
------------------------
- Upload file
- Download file
- Sync files across devices


Non-functional requirements:
----------------------------
- Availability over consistency
  - Download the older version of
    the file instead of waiting
    for a new version to propagate
    is ok
- low latency for uploads and downloads
- support large files (50Gb)
  - resumable uploads
- high data integrity (accurate syncs)


Core entities:
--------------
- File (row bytes)
- File Metadata
- Users


API:
---------------
- POST /uploadFile({metadata}) --> {upload_id, presigned_url}

- POST /getFile({file_id}) --> {file{}, metadata}

- POST /getChanges({since}) --> file_metadata[]

Functional requirements:
------------------------
- Scrape the web starting from seed URLs
- Extract and store text data


Non-functional requirements:
----------------------------
- Fault-tolerant
- Politeness (respect robots.txt, don't DDoS)
- Scale to 1B pages
- Efficient (crawl within a week)


Core entities:
--------------
- URLs
- Text data
- Domain metadata


Interface:
----------
- input: seed URLs
- output: text data


Data Flow:
----------
1. Take seed URLs from a frontier and the IP from DNS
2. Fetch HTML
3. Extract text
4. Store text in the database
5. Extract URLs from the page and add then to the frontier
6. Repeat steps 1-5 until all URLs are crawled