Python script to parse through an S3 folder(including subfolders) and get the list of all its objects in a CSV file.

Requirement was to create a s3 manifest file for s3 batch job operation. More details around s3 manifest file can be found here. import boto3 from datetime import datetime import csv import logging import argparse today = datetime.today().strftime('%Y-%m-%d') now = datetime.today() print(now) logging.basicConfig(filename='backup_manifest_file_generation_'+today+'.log', level=logging.INFO) rows = [] larger_files = [] def get_matching_s3_keys(bucket, prefix='', suffix=''): """...

Delete AWS EC2 Snapshots and AMIs older than certain no. of days – Python

Businesses are using the AWS cloud to enable faster disaster recovery of their critical IT systems without incurring the infrastructure expense of a second physical site. The AWS cloud supports many popular disaster recovery (DR) architectures from “pilot light” environments that may be suitable for small customer workload data center failures to “hot standby” environments...

AWS LAMBDA – ssh into a remote EC2 linux machine using Python ‘paramiko’ package module.

AWS Lambda can be used to connect to remote Linux instances by using SSH and run desired commands and scripts at regular time intervals. This is especially useful for scheduling tasks on your EC2 instances, when you don’t want to manage cron or external schedulers for a dynamic fleet of instances. Here, I will be...

AWS CloudFormation template to create an EMR.

Using the below template you can create a sample EMR with 1 master node and 2 core nodes of type 'm3.xlarge'. The EMR name and tag values are passed as parameters which will enable you to provide the same during the template execution. Here, I have provided only two application to be installed in the...

ERROR: — | The encryption key has exceeded its grant limit in AWS KMS |

_LimitExceededFault :: As Error a => Getting (First ServiceError) a ServiceError _LimitExceededFault =     _ServiceError . has Status 400 . has Code "LimitExceededFault" Solution: Basically there is a limit on “Grants for a given principal per CMK” which is 30 i.e., no more than 30 grants can specify the same grantee principal. For ex: in case...

Drop user in AWS Redshift/RDS(PostgreSQL).

As of today, Redshift does not have a REASSIGN functionality and you cannot drop a user if the user - is the owner of an object; and/or. has some privilege on any object. You can follow below steps to to revoke all the privileges assigned to that user and later drop user from the database...

Python script to remove the default VPC of all the regions in an AWS account.

I never found a good description on how to delete a default VPC programmatically. So I have come up with a Python script that attempts to delete those pesky default VPCs in all regions from your AWS account. Warning: Deleting the default VPC is a permanent action. You must contact AWS Support if you want...

AWS Key Management

How to list/revoke the grants attached to a resource for a particular KMS key? Basically there is a limit on "Grants for a given principal per CMK" which is 30 i.e., no more than 30 grants can specify the same grantee principal. For ex: in case of redshift in us-east-1 region, the grantee principal is ‘redshift.us-east-1.amazonaws.com’....