DEV Community

Cover image for Fitter - open-source instrument for amazing scraping.
Iurii Panarin
Iurii Panarin

Posted on

Fitter - open-source instrument for amazing scraping.

Hello everyone!

I want to introduce Fitter for you.

GitHub logo PxyUp / fitter

New way for collect information from the API's/Websites

Fitter + Fitter CLI

Fitter - new way for collect information from the API's/Websites

Fitter CLI - small cli command which provide result from Fitter for test/debug/home usage

Fitter Lib - library which provide functional of fitter CLI as a library

Way to collect information

  1. Server - parsing response from some API's or http request(usage of http.Client)
  2. Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information
  3. Static - parsing static string as data

Format which can be parsed

  1. JSON - parsing JSON to get specific information
  2. XML - parsing xml tree to get specific information
  3. HTML - parsing dom tree to get specific information
  4. XPath - parsing dom tree to get specific information but by xpath

Use like a library

go get github.com/PxyUp/fitter
Enter fullscreen mode Exit fullscreen mode
package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse
Enter fullscreen mode Exit fullscreen mode

This tool can do a lot of stuff:

  1. Read content by XML/XPath/Json/HTML selectors
  2. Emulate browser with docker/playwright/chromium
  3. Support proxy
  4. Global references
  5. Fallback on no empty fields
  6. Notification Telegram/Http/Redis/Console
  7. File downloading
  8. Full README.md
  9. Can work as golang library
  10. Support runtime golang plugins
  11. Work as CLI
  12. Works on Linux/Mac/Windows.
  13. Support different type of fields
  14. Support expression language for field and notification settings

......

N+1. Best and most important it is gluing of request

Repository have a multiple examples for CLI/Fitter

Real world example of the problem

Last week my friend need to monitor pool on some crypto stock. So here his problem:

  1. First of all you need get all user positions https://docs.saucerswap.finance/v/developer/saucerswap-v2/liquidity-operations/get-user-positions
  2. After that all stock pool https://docs.saucerswap.finance/v/developer/saucerswap-v2/liquidity-operations/fetch-all-pools
  3. After that you should check if pool has user positions you should send notification.

Let's see how Fitter can solve this problem:

Let me explain what is here:

  1. environment variables:

TELEGRAM_TOKEN - line 95

USER_ID - line 37

So my friend need just run Fitter and add UserIds for Telegram notification(line 95):

TELEGRAM_TOKEN=xxx USER_ID=xxx ./fitter --path=./config.json
Enter fullscreen mode Exit fullscreen mode
  1. What happens in config:

2.1 Creating Reference(line 3) on all pools and caching on 60 seconds(line 4). We take the response and create an array of the objects with two fields tokenA and tokenB.

2.2 Creating Reference(line 33) on AccountId will be just string from ENV.

2.3 We create items for monitoring every 10 seconds(line 88) which take values from the server via injecting AccountIdRef(line 52). Response it is array of objects with two field: first it is (line 66) - template string with injecting token name; second is boolean field which show if pair in pool(line 71)

  1. Next we will inform if some of the values in array is true(line 93).

We will receive notification in Telegram:

[
 {
  "position": "tokenA=USDC tokenB=USDC[hts]",
  "value": true
 },
 {
  "value": false,
  "position": "tokenA=USDC tokenB=USDT"
 }
]
Enter fullscreen mode Exit fullscreen mode

Top comments (0)