Terraform State Nightmare

If you've used terraform, you know how important the tfstate file is. In case you're taking it lightly do not(I'm serious)

Today my colleague and I were working on creating an S3 backend, but due to a Resource Error which was probably not caused by the state file we tried to do a terraform init but accidentally did a terraform init -reconfigure, instead of a terraform init -migrate-state(Mistakes happen, we're humans). Luckily the state file was backed up. Now I want to also document the steps taken here and what failed.

Disclaimer: I'm new to terraform, the following is my personal experience.

• We tried to upload the state file to S3 and tried to do terraform init -migrate-state(Spoiler Alert: It didn't work). Terraform just closed its eyes and was like I don't see the resources.
• We tried switching back to local backend, but the statefile there too didn't work, we even replaced .terraform/terraform.tfstate
• When we did a terraform apply -state=backed.tfstate it did recognize our resources, so we knew there was hope.
• Apparently terraform init doesn't have a -state option...
• Somewhere along the line we started getting the error

Terraform 0.11.1 does not support state version 4, please update

The solution? Hidden in a Stackoverflow answer.

Terraform does not support state version 4 is a common issue.

There is only one fix to this

Take the local state file and push it to some remote backend and then delete the .terraform directory and run the init command with the backend.
It should work

Frustration had hit the roof, we'd been at this for almost an hour or more(Time goes very fast when you're debugging)

I think I was at the anger stage. Everything we tried just gave us no results and terraform was acting as if it couldn't see the state file.

Then we had the terraform state file so we did a
terraform state push backup.tfstate and lo-and-behold, we got some random lineage error.

Modified the lineage parameter in the tfstate file to match the file which terraform created in the s3 bucket. And that was it. Terraform agreed that the file was up to its standards and it got all the resources intact.

Takeaways from this.

• Always enable object versioning in your S3 Bucket or object storage.
• Always backup your state files.
• Always try all sort of combinations when it comes to terraform, there is a workaround hidden almost somewhere.

Thanks for reading, hopefully the Terraform spirit is never angry on you.