Motivation
Tracking application errors should be a check list item for all production apps. In simple cases, a automatic "email the stacktrace to admin" is suffice to start but reaches the limit very fast. Items like
- Javascript Error Tracking, (in 2020 with source maps),
- Environment information, like logged in user,
- combining, ignoring, merging errors,
are requirements most growing projects will have sooner or later. Having an error tracking system in place is the best solution.
For our case, we prefer to self-hosted solutions like this, which has significant advantages:
- no additional data protection issues, or signing additional data processing agreements, as all the data remains in the company's sphere of influence.
- integration with company systems, like VPN/SAML usually more easy or is even only possible by self hosting,
- sometimes being the only user of your systems provides performance benefits , e.g. comparing with using Sentry's hosted version, our current installation seems to be orders of magnitude faster (experienced response times like 5-10 seconds when using Sentry free tier), also because the error tracking and app servers are not only in the same country but maybe even the same DC like your app which drastically reduces latency (Sentry) or transport cost when using AWS.
- saving some money per month instead of paying "per app"
On the downside, running things themselves can be more risky, especially when a Error tracking system is hard to setup.
From the very beginning, we used Errbit error tracking, which provides a Airbrake compatible API. But having tried out Sentry.io for private projects before, I was blown away by the features, like tons of plugins, beautiful clean UI, top notch Javascript integration with breadcrumbs and source maps, so I've decided to set it up in a self-hosted manner for our company. Fortunately, Sentry provides a Docker installation script, which I will use during this guide.
Installation of self hosted Sentry
First, we set up a Cloud VPC. Size of the cloud instance depends on your prospected error volume. We started with a Hetzner CX21 which costs about 5 EUR per month and can be upgraded easily via button click and rescaled later on, if necessary.
After starting the instance, install Docker and Docker compose on the host, or do it like us, and use a cloud-config yaml during installing (Pasting into field user config on Hetzner Cloud), e.g.:
# cloud-config
apt:
sources:
docker.list:
source: 'deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable'
keyid: 0EBFCD88 # GPG key ID published on a key server
packages:
- apt-transport-https
- bash-completion
- ca-certificates
- command-not-found
- curl
- debian-archive-keyring
- dnsutils
- fail2ban
- docker-ce
- docker-compose
- golang-go
- git-core
- htop
- lshw
- lsof
- ltrace
- make
- software-properties-common
- sysstat
- tar
- unattended-upgrades
- vim
After powering up, clone this repository to /opt/sentry
$ git clone https://github.com/getsentry/onpremise /opt/sentry
# Or, use this fork that has Caddy/Letsencrypt support
$ git clone https://github.com/merantix/sentry /opt/sentry
This setup will include everything you need BUT a HTTPS proxy. To have Caddy run as an HTTPS Proxy with Auto-Letsencrypt, check out this fork merantix/sentry.
Adjustments and configuration
If you are wanting to run Sentry in a organisation setting, you might want to install some plugins. In our case, these are a Mattermost plugin and a SAML2 plugin for user authentication.
Add those requirements to the Dockerfile
in the getsentry folder:
# /opt/sentry/Dockerfile
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE}-onbuild
+ RUN pip install https://github.com/getsentry/sentry-auth-saml2/archive/master.zip
+ RUN pip install -e git+https://github.com/NDrive/sentry-mattermost@master#egg=sentry-mattermost
+ # Or instead, a forked version with Mattermost multi channel support
+ RUN pip install -e git+https://github.com/zealot128/sentry-mattermost.git@merged#egg=sentry-mattermost
(If you are using Mattermost, and like to reuse webhooks between multiple channels/projects, I've recommend my fork until this PR is merged)
Now, check out configuration in docker-compose.yml
and env
. In general, we had very little configuration:
- SENTRY_SECRET_KEY in
env
(or will generate during installation) - changed Caddy hostname in
Caddy/Caddyfile
, make sure, that hostname resolves to the server's ip for letsencrypt - set, or remove email settings in docker-compose.yml or env
If you are finished, run:
# will ask for a start password for the admin user
$ ./install.sh
# start all the services
$ docker-compose up
Some quick docker-compose commands you might need later on:
# show stdout of a host
$ docker-compose logs web
# enter a host
$ docker-compose exec web bash
# start/stop a host
$ docker-compose restart web
$ docker-compose up web
$ docker-compose down web
SAML
If you are using a SAML2 provider for your organisation, you can try to add the SAML authentication next. The specific settings are not standardized, and almost all supplier use different kind of names for the person's attributes (Claims). For example, we are using a (of course self-hosted) custom SAML2 server based on Ruby-SAML by Onelogin and also added several extra fields via a OID extra.
Those settings are based on these attributes (claims) by Ruby SAML, but like I said, that depends on your provider:
Debugging attribute mapping
To find out the specific names of the mapping, that you can use, you can add a print statement into the right Django file. I've gone this way:
$ docker-compose exec web bash
$ apt install vim
$ vim /usr/local/lib/python2.7/site-packages/sentry/auth/helper.py
Add import pprint
somewhere in the top of the file, and add a printf statement in the finish_pipeline
function:
+ import pprint
# ...
def finish_pipeline(self):
data = self.fetch_state()
# The state data may have expried, in which case the state data will
# simply be None.
if not data:
return self.error(ERR_INVALID_IDENTITY)
try:
+ pprint.pprint(data)
identity = self.provider.build_identity(data)
Exit the docker container and restart web docker-compose restart web
. Now try set up of auth again and watch docker-compose logs web
for debugging output.
Mattermost
Mattermost integration was straight forward:
- Create a webhook as a Mattermost admin, copy the API url
- Mattermost must be enabled as a Legacy Integration per project, after enabling add Webhook url.
- My fork of the plugin also supports changing the channel per project and reusing the Webhook.
Open Source used:
getsentry / self-hosted
Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept
Self-Hosted Sentry
Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept.
Documentation here.
Official bootstrap for running your own Sentry with Docker.
Requirements
- Docker 19.03.8+
- Compose 1.24.1+
Minimum Hardware Requirements:
- You need at least 2400MB RAM
Setup
To get started with all the defaults, simply clone the repo and run ./install.sh
in your local check-out.
During the install, a prompt will ask if you want to create a user account. If you require that the install not be blocked by the prompt, run ./install.sh --no-user-prompt
.
There may need to be modifications to the included example config files (sentry/config.example.yml
and sentry/sentry.conf.example.py
) to accommodate your needs or your environment (such as adding GitHub credentials). If you want to perform these, do them before you run the install script and copy them without the .example
extensions in the name (such as sentry/sentry.conf.py
) before running the install.sh
script.
The recommended way to customize your configuration is using the files…
getsentry / sentry-auth-saml2
SAML2 SSO provider for Sentry
SAML2 Auth for Sentry
DEPRECATED: This project now lives in sentry
Note: SAML2 Authenttication is still currently an experimental feature.
An SSO provider for Sentry which enables SAML SSO and SLO support, including various identity provider support.
The following identity providers are supported
A generic SAML2 module is also provided, which may be configured with any Identity Provider that conforms to the SAML2 specification.
Install
$ pip install https://github.com/getsentry/sentry-auth-saml2/archive/master.zip
Configuration
Refer to the Sentry Single Sign-On documentation for individual SAML Identity Provider configurations.
Refer to the Enabling SSO documentation for what feature flags to enable for this plugin.
NDrive / sentry-mattermost
Sends Sentry notifications to Mattermost Open Source Chat
Sentry Mattermost
A plugin for Sentry to enable notifications to Mattermost Open Source Chat. This is based in the sentry-slack plugin: https://github.com/getsentry/sentry-slack
Usage
Install with pip and enable the plugin in a Sentry Project:
pip install sentry_mattermost
Configure Mattermost:
- Create an Incoming Webhook
- Enable override usernames and profile picture icons in System Console Integrations
Contributing
We use Docker to setup a development stack. Make sure you have the latest Docker Toolbox installed first.
First time setup
Setups Docker containers and Sentry admin:
make bootstrap restart
Development
Each time you update the code, restart the containers:
make restart
And access the sentry admin at
http://<DOCKER IP>:8081
Top comments (1)
Hi Stefan,
Did you used this mattermost integration in version higher than 9+?