So we track our error logs via Sentry on-premises which we hosted on our own. Recently it started to give error, the disk space is running out!!!
WHY??
Apparently we needed to clear the database after certain time. Otherwise it will stack up all of the error logs.
Solution??
We log into our server. Sentry was running in docker. We went to the docker folder ran this
docker-compose exec worker bash
After that from the worker bash we ran
sentry cleanup --days 30
basically this will clean up all the events data before 30 days.
After this we go inside of the database by running these
docker-compose exec postgres bash
psql -U postgres
-
VACUUM FULL;
Point to be noted,VACUUM FULL;
will lock your db tables unless the full vacuum is being done.
Voila! Database and hard driver storage cleaned up! :)
Top comments (4)
My solution was used before I found this pull request.
To checkout volume:
To connect to docker:
To enter console:
To find out all tables with TOAST tables:
To cleanup:
PROFIT.
why a "truncate" after the delete ?
I was wondering why the disk size wasn't shrinking, turned out while running VACUUM nodestore_node; it returned pq: could not resize shared memory segment "..." to ... bytes: No space left on device.
the solution to this was found here stackoverflow.com/questions/567515...
I just added shm_size: '1gb' to the postgres docker definition and rerun docker compose up that recreated the postgres container. after this, the vacuum started working correclty :)
Hello Ashraful,
For me I am stuck at this in worker container. I have close to 1.1 TB data here.
root@6868e8f50bf8:/# sentry cleanup --days 8
/usr/local/lib/python3.8/site-packages/sentry/runner/initializer.py:555: DeprecatedSettingWarning: The SENTRY_URL_PREFIX setting is deprecated. Please use SENTRY_OPTIONS['system.url-prefix'] instead.
warnings.warn(DeprecatedSettingWarning(old, "SENTRY_OPTIONS['%s']" % new))
/usr/local/lib/python3.8/site-packages/memcache.py:1303: SyntaxWarning: "is" with a literal. Did you mean "=="?
if key is '':
/usr/local/lib/python3.8/site-packages/memcache.py:1304: SyntaxWarning: "is" with a literal. Did you mean "=="?
if key_extra_len is 0:
06:03:37 [INFO] sentry.plugins.github: apps-not-configured
Removing expired values for LostPasswordHash
Removing expired values for OrganizationMember
Removing expired values for ApiGrant
Removing expired values for ApiToken
Removing expired files associated with ExportedData
Removing old NodeStore values
It is stuck here for about 15 mins now. Any idea what can be done now?
Also, here is how psql looks
postgres=# SELECT oid::regclass, reltoastrelid::regclass, pg_relation_size(reltoastrelid) AS toast_size FROM pg_class WHERE relkind = 'r' AND reltoastrelid <> 0 ORDER BY 3 DESC;
oid | reltoastrelid | toast_size
--------------------------------------------+-------------------------+--------------
nodestore_node | pg_toast.pg_toast_20250 | 561381097472
pg_rewrite | pg_toast.pg_toast_2618 | 385024