Jq to filter aws describe-instances json output

Assuming /tmp/instances contains a list of instance ids:


for i in `cat /tmp/instances`

do 
    echo $i
    aws --profile=awsdev --region=eu-west-1 ec2 describe-instances --instance-id $i \
    | jq '.Reservations|.[]|.Instances|.[0]|.Tags[]|select(.Key=="Name")|.Value'

done

Advertisements

AWS Firehose Lambda template for vpc-flow logs to ELK cluster

When creating a Kinesis Firehose Stream, some Lambda blueprints are available to process and convert data.

https://eu-west-2.console.aws.amazon.com/lambda/home?region=eu-west-2#/create?tab=blueprints

Using the Python blueprint:

kinesis-firehose-cloudwatch-logs-processor-python

 

To add a timestamp in front of it, modify the transformLogEvent(log_event) function like this:

# return log_event['message'] + '\n'
ts = str(log_event['timestamp'])
return ts + ' ' + log_event['message'] + '\n'

Python aws boto3 print instances list per vpc


import boto3

session = boto3.Session(profile_name='awsdev')
ec2= session.client('ec2', region_name='eu-west-1')
ec2.describe_instances()

response=ec2.describe_instances()

# get vpc list
vpcs=[]
for i in range(len(response["Reservations"])):
    instance_vpc=response["Reservations"][i]["Instances"][0]["VpcId"]
    vpcs.append(instance_vpc)
vpcs=list(set(vpcs))

# dict with k=vpc v=[instances]
dict_vi={}
for v in vpcs:
    dict_vi[v]=[]
    for i in range(len(response["Reservations"])):
        instance_vpc=response["Reservations"][i]["Instances"][0]["VpcId"]
        instance_id=response["Reservations"][i]["Instances"][0]["InstanceId"]
        if instance_vpc == v:
            dict_vi[v].append(instance_id)

# print vpc and instance list
for i in dict_vi:
    print("---------------------- %s ---------------------" % i)
    print(', '.join(dict_vi[i]))
    print('\n')


Python3 argparse

https://docs.python.org/3/howto/argparse.html

Supports -h and --help automatically :D

So you automagically get this:

~$ ./myprog -h
usage: myprog [-h] [--cron] [--ms MS]

optional arguments:
-h, --help show this help message and exit
--cron use when you want to lock i.e. when run by cron
--ms MS overwrite milliseconds treshold in local_config.py

See example snippet to use booleans switches or pass values:

import argparse

# parse cli args
parser = argparse.ArgumentParser()
parser.add_argument("--cron", action="store_true", help="use when you want to lock i.e. when run by cron")
parser.add_argument("--ms", type=int, help="overwrite milliseconds treshold in local_config.py")
args = parser.parse_args()

if args.ms:
    ms_treshold = args.ms

Python import csv file with support for comments in dictionary


import csv
from collections import OrderedDict

# Import csv as list

with open('owners.csv', 'r') as owners:
reader = csv.reader(filter(lambda row: row[0]!='#', owners))
o = list(reader)

# convert list into dict with k=name[0] v=[rest of list]
od={}
for i in o:
    try:
        od[i[0]]=i[1:]
    except:
        pass

odsorted=OrderedDict(sorted(od.items(), key=lambda t: t[1]))

"""
example csv file:
~$ cat owners.csv
# comments supported :)
name1,owner1
name2,owner2

to access dict (i.e. in ipython):
In [1]: od["name1"][0]
Out[1]: 'owner1'
In [2]: odsorted["name1"][0]
Out[2]: 'owner1'
"""

Python script to compare two terraform resource files

#!/usr/bin/env python3

import sys, re

iam_users = open("iamu.tf").read()
matches_iam_users = re.findall(r'(resource.*?})', iam_users, re.MULTILINE + re.DOTALL)

iam_users_to_remove = open("iam-users_to-remove.tf").read()
matches_iam_users_to_remove = re.findall(r'(resource.*?})', iam_users_to_remove, re.MULTILINE + re.DOTALL)

matches_users_to_keep = [item for item in matches_iam_users if item not in matches_iam_users_to_remove]

with open("iamu_keep.tf",'a') as users_to_keep:
users_to_keep.write('\n'.join(matches_users_to_keep))