DEV Community

Cover image for Deleting a Lambda leaves its log group behind. What do you do with the ones nobody owns?
Gonzalo Gjuka
Gonzalo Gjuka

Posted on

Deleting a Lambda leaves its log group behind. What do you do with the ones nobody owns?

This week I went through an AWS account where Lambda functions come and go every sprint. The functions were gone. Their log groups were not. CloudWatch creates /aws/lambda/<name> on the first invoke with retention set to "Never expire", and deleting the function does not touch it. The Lambda docs say it in one sentence; nobody reads that sentence until the bill does.

The count was not dramatic in dollars. A few hundred GB of logs nobody will ever open, at the storage rate that is a few dollars a month. The part that bothered me was the ownership. The team that wrote those functions moved on, the Terraform that created them never declared the log group, and the destroy that removed the function left the group with no stack, no tag and no owner. It sits in the list between groups that matter.

AWS added account-level retention policies this month, so now one put-account-policy with a /aws/lambda prefix puts an expiry on all of them, existing ones included. That stops the growth. It does not delete the group, and it does not tell you which ones still have a function behind them.

My current answer is a cross: list the groups with no retention, list the functions that exist, and treat the difference as orphans to delete after a snapshot of the names. It works, and it is a script I have to remember to run.

So the question for people running more accounts than I do: do you delete orphaned log groups at all, or do you set retention and let them fade to zero bytes? And if you delete, what stops you from deleting the one that a compliance rule wanted kept?

Top comments (6)

Collapse
 
iictp profile image
IICTP •

The cross-reference approach you describe is the one I'd trust most — listing groups with no retention against the functions that still exist makes the orphan set explicit instead of guessed. I'd still lean toward deleting after the snapshot rather than letting retention fade them out, since a group with no owner and no retention is exactly the kind of thing that quietly accumulates. Do you store the snapshot somewhere the team can find it later, or is it more of a one-off record before the delete runs?

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

One-off, but one that ends up in git. Before the delete runs, the script writes the describe-log-groups output for the candidates (name, storedBytes, creationTime) to a dated file in the same ops repo, and the PR that ran it is the record. Enough so far: the question that comes back later is "what did we delete and when", not "give me the bytes". If someone wants the content, create-export-task to S3 before delete-log-group covers it.

I agree that fading them out is the wrong default. The guard I keep on the delete is list-tags-for-resource per candidate: anything tagged retain, or still in a Terraform state with skip_destroy, gets skipped and listed in that same file. A tag is the one thing a compliance rule can leave on a group with no owner.

Collapse
 
iictp profile image
IICTP •

That answers it neatly — a dated file in the ops repo, with the PR as the record, covers "what did we delete and when" without keeping the bytes around forever. I especially like the guard: checking list-tags-for-resource for retain and skip_destroy before deleting turns a cleanup script into something a compliance rule can actually trust. Have you ever had a group survive that check and later turn out to be genuinely needed, or does the skipped list mostly stay quiet?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Not yet. The skipped list has stayed short and quiet, and nothing on it has turned out to matter the other way either. The near miss I had was in the cross-reference, not in the guard: Lambda@Edge writes to /aws/lambda/us-east-1.<name> in every Region where the function ran, while the function itself only exists in us-east-1, so a per-Region list of "groups with no function" flags all of those as orphans. Since then the script also reads describe-log-streams --order-by LastEventTime --descending --limit 1 per candidate, and anything with an event in the last 30 days is not an orphan no matter what the function list says. Empty on both counts is the only thing that gets deleted.

Collapse
 
mariosfong profile image
Mario Fang •

Same problem here, just with old SQL Server jobs and scheduled tasks instead of Lambda. What finally worked was making ownership a required attribute rather than a cleanup project: anything without an owner tag goes on a list, gets exported to cold storage with a name and date range manifest, then gets deleted 30 days later unless someone claims it. The export is what saves you in an audit, not the tag.

For the compliance side I keep a small allowlist file in git naming the resources we must keep, and the delete script skips them by name. Tags get edited without a trace. A file in git has a commit history and a reason attached to it.

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

The export-then-wait-30-days part is the piece most people skip, and I agree that is what saves you in an audit. One thing I would push back on for AWS: a tag edit on a log group is not traceless. CloudTrail records UntagResource and DeleteLogGroup for CloudWatch Logs, so who removed the tag and when is there. What a tag cannot carry is the reason, and that is where the file in git wins. Where I ended up is both: the allowlist in git names what must stay, and the tag is what the script reads at run time, because the file does not travel to the account. For new functions the fix is upstream: LoggingConfig.LogGroup lets the function log to a group Terraform created with retention and tags, instead of the one Lambda creates on first invoke.