Started my Project on Energy-Efficient DevOps: Auto-Suspending Idle AWS Resources using ML

Week 2 – Rebuilding Infra with Terraform + S3 Backups

Objective:

This week, I focused on phasing out the old manually created EC2 instances and moving to a cleaner, automated setup using Terraform and Ansible. But before wiping anything, I made sure to back up all logs, models, and scripts to S3 to avoid losing any data from earlier experiments.

What I Did This Week:

Backed Up Everything from Old EC2

Before tearing anything down, I used Ansible to upload important files from the original devops-node-1 instance to an S3 bucket:

cpu_log.csv, model.pkl, and workload_log.txt
All .py, .sh, and .txt scripts related to ML predictions and simulations

Bucket used: energy-efficient-sahil

Set Up Terraform on Windows

Installed Terraform CLI and configured AWS CLI with aws configure
Wrote a basic main.tf to spin up a t2.micro EC2 instance with:
- Ubuntu AMI
- IAM role (ec2stopaccessrole) with EC2 + S3 permissions
- Security Group + key pair setup

After some .tf.txt confusion and credential errors, everything finally clicked into place.

Provisioned a Clean EC2 Instance

Ran terraform apply to spin up the new instance
Verified SSH access
Made sure the IAM role was correctly attached so no NoCredentialsError in boto3

Migrated Project Files to New Node

Transferring files wasn’t smooth at first. SCP from EC2 to EC2 failed because .pem files aren’t accessible inside instances. So I:

Downloaded files to my laptop from the old EC2
Re-uploaded them to the new Terraform-managed instance using PowerShell + SCP

All Commands I Ran (and Why)

Command	Purpose
`ansible-playbook upload_to_s3.yml`	Backup data to S3
`terraform init` / `plan` / `apply`	Deploy new EC2
`aws configure`	Set up AWS access keys
`scp` from laptop	Move files between instances
`terraform output`	Grab EC2 IP for Ansible inventory

What Went Wrong — and How I Fixed It

Issue	Fix
Terraform said “no config files”	Renamed `main.tf.txt` to `main.tf`
Terraform couldn’t find AWS creds	Ran `aws configure` again
Ansible wouldn’t upload everything	Fixed `with_items` list
`scp` from EC2 didn’t work	Used local machine instead
S3 bucket creation failed	Updated IAM policy to allow `s3:CreateBucket`

Current Status

S3 bucket contains all important logs, models, and scripts
New EC2 instance provisioned using Terraform
Old files migrated successfully
IAM roles, metadata access, and boto3 are all working
Old EC2s (devops-node-1, devops-node-2) still need to be deleted

Want to Follow Along?

I’ll be sharing weekly progress — issues, logs, architecture, and ML models.

If you've solved similar problems (like automated cloud optimization), I’d love to hear your insight.