My Workflow

I spent a few nights wrapping Google's analyzeSentiment API into a GitHub Action. The Action runs Sentiment Analysis over the content of HTML files and provides an overview of the overall emotion of all (the selected) pages in your project.

The API returns values from -1 to 1, indicating how strong a certain emotion – positive or negative – is. After running the Action, a table with the score per each page is printed in its logs. Read more about Interpreting sentiment analysis values.

Along with other content analysis tools, it might come in handy to maintainers who want to understand the text that is pushed to the project every day. See it in action 🚀

⚠️ Now I got really excited about this and will continue developing along with other automation ideas that I have. This being an early release of the Action, please take a look at the roadmap and submit issues with what kind of features you would like to see in future releases.

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

Here is an example of how to use the Action on public .html files.

name: Sentiment analysis on public

on: push

jobs:
  analysis:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2 #Be sure you checkout the files beforehand
    - name: Run sentiment analysis on HTML files
      uses: bogdaaamn/copy-sentiment-analysis@v0.6.1
      with: 
        gcp_key: ${{ secrets.GCP_KEY }} #Google Cloud Platform API key. Read the README for instructions 

Along with the code – more examples, requirements and a known issues roadmap are available in the bogdaaamn/copy-sentiment-analysis repository (view it on Marketplace).

name: Sentiment analysis on public

on: push

jobs:
  analysis:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2 #Be sure you checkout the files beforehand
    - name: Run sentiment analysis on HTML files
      uses: bogdaaamn/copy-sentiment-analysis@v0.6.1
      with: 
        gcp_key: ${{ secrets.GCP_KEY }} #Google Cloud Platform API key. Read the README for instructions 

Additional Resources / Info

Open source use cases

At the moment, the overview table is printed in the Actions tab, after the code is running. But it seems counter-intuitive and having too much friction in between.

I am curious about what the community thinks: How would you see this Action printing the results? A comment to the PR? A table in Action's log? Failing if there are too many negative results?

⚠️ GCP's bias in sentiment analysis

A few years ago, Google API was criticized in the media for producing bias results towards race, gender, and religion. So I had mixed feelings about using a pre-trained model in this Action.

Now, it is hard to understand what is going on with Google's proprietary algorithm and how they fight unwanted bias, but more recent research (see charlescearl, 2019) concluded that GCP seems to be less sensitive to the race or gender of participants than other competitor platforms. The same article recommends that users should proceed with caution and conduct evaluations on their own. The tests that I've done had neutral results, but I am ready to expand my research and pull the plug if needed.

Moreover, there is extraordinary research done towards identifying, analyzing, and diminishing bias in data (see Dixon et al., 2018 from Google Research or Caliskan et al., 2016 and May et al., 2019) and all I hope is that Google (or really any cloud provider out there) is doing better and better. I believe it is really important to enable and support bias and fairness research – especially now, after the recent upbringing of GPT-3 (see Burus, 2020) when the society gets exposed more and more to technology.

Sources

• Vice: Google’s Sentiment Analyzer Thinks Being Gay Is Bad

• Engadget: Google's sentiment analysis API is just as biased as humans

• Techleer: Google Sentiment Analysis API gives a biased output

• Gender and Racial Bias in Cloud NLP Sentiment APIs

• Measuring and Mitigating Unintended Bias in Text Classification

• Semantics derived automatically from language corpora contain human-like biases

• On Measuring Social Biases in Sentence Encoders

• The (Un)ethical Story of GPT-3: OpenAI’s Million Dollar Model