Troubleshooting Datadog Log Setup

Description

The setup of datadog is so simple for anybody to do. But to get it to ship logs can be messy. This guide would help you in troubleshooting.

Known Issues

After installation and having all configurations right, this include logs_enabled: true set in your datadog.yaml, your datadog log setup may still not work.
These lessons are from experience and you can refer to this in times of need.

1. Datadog Log

The first point of call is datadog logs. note: These logs are not managed by Logrotate so it can get very huge. I advise you tail -n 500 /var/log/datadog/agent.log.

Check this log for TCP(Port) and Permission issues.

sudo cat /var/log/datadog/agent.log
# or
sudo cat /var/log/datadog/agent.log | grep permission

1. Users and Usergroup

From experience datadog works well if the /etc/datadog/conf.d/awesome-app.d/conf.yaml is owned by
dd-agent user and group you created during installation.
If you are not sure, run cd /etc/datadog/conf.d/awesome-app && ll and view the usergrp and user. If the directory is owned by root or any other user, you can fix it with the following.

sudo chown dd-agent awesome-app.d/conf.yaml
sudo chgrp dd-agent awesome-app.d/conf.yaml

# and restart datadog

sudo service datadog-agent restart

Verify if you are still unable to ship logs.

1. Grant Permission to Datadog

Most of the time, we can get lucky by running some chmod on the log file we are shipping to datadog. This is not so good since the directory and log is generated by an application with a different user and group.

So the best approach here is setfacl. It grants permission on a file without the need to compromise its previous users or permission.

Execute it as below per your path

setfacl -m u:dd-agent:rwx /var/log/awesome-app/file.log

or grant it on a directory

setfacl -m u:dd-agent:rwx /var/log/awesome-app

Restart and you would be fine.

Visit https://docs.datadoghq.com/ for more!