How to Find Unused Amazon EC2 Security Groups

Question

I am trying to find a way to identify orphaned security groups so that I can clean and get rid of them. Does anyone know a way to detect unused security groups.

It will work either through the console or using command-line tools (launching command-line tools on Linux and OSX machines).

Answer 1

Note. This applies only to the use of security in EC2, and not to other services such as RDS. You will need to do more work to include the security groups used outside of EC2. It’s good that you cannot easily (possibly even impossible) delete active security groups if you missed one related w / other service.

Using the new AWS CLI tool, I found an easy way to get what I need:

First get a list of all security groups.

aws ec2 describe-security-groups --query 'SecurityGroups[*].GroupId' --output text | tr '\t' '\n' 

Then get all the security groups bound to the instance, then go to sort , then uniq :

 aws ec2 describe-instances --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text | tr '\t' '\n' | sort | uniq 

Then put it and compare 2 lists and see what is not used from the main list:

  comm -23 <(aws ec2 describe-security-groups --query 'SecurityGroups[*].GroupId' --output text | tr '\t' '\n'| sort) <(aws ec2 describe-instances --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' --output text | tr '\t' '\n' | sort | uniq) 

Answer 2

If you select all your security groups in the EC2 console, then click actions → Delete security groups, a pop-up window will appear informing you that you cannot delete security groups attached to instances, other security groups or network interfaces, and in it will list the security groups that you can delete; those. unused security groups :)

Answer 3

This is sample code written in boto (Python SDK for AWS) to list the security group by the number of instances it is associated with.

You can use this logic to get the same thing on the command line

Code Boto

 import boto ec2 = boto.connect_ec2() sgs = ec2.get_all_security_groups() for sg in sgs: print sg.name, len(sg.instances()) 

Exit

 Security-Group-1 0 Security-Group-2 1 Security-Group-3 0 Security-Group-4 3 

Answer 4

After about a year of unaudited use, I found it necessary to audit my AWS EC2 security groups and clean out obsolete unused groups.

It was a difficult task to do through the web interface, so I turned to the AWS CLI to make the task easier. I found a start on how to do this in StackOverflow, but it was far from complete. So I decided to write my own script. I used AWS CLI, MySQL and some "bash-foo" to do the following:

• Get a list of all EC2 security groups. I store the group id, group name and description in the declared "groups" in a MySQL database called aws_security_groups on the local host. The total number of found groups is reported.

• List all security groups associated with each of the following services and exclude them from the table: EC2 Istances EC2 elastic balancing loads AWS RDS instances AWS OpsWorks (should not be deleted on Amazon) Default security groups (cannot be deleted) ElastiCache

For each service, I report the number of groups remaining in the table after the completion of the exception.

3. Finally, I show the group id, group name and description for the remaining groups. These are “unused" groups that need to be checked and / or deleted. Ive found that SG between instances and Elastic Load Balancers (ELBs) often refer to each other. It’s best to do a manual exploration to make sure that they aren’t actually used until you remove cross-references and remove security groups. But my script at least views this as something unacceptable.

NOTES: 1. You will need to create a file to store your host, MySQL username and password and specify the $ DBCONFIG variable for it. It should be structured as follows:

 [mysql] host=your-mysql-server-host.com user=your-mysql-user password=your-mysql-user-password 

2. You can change the database name if you want - be sure to change the $ DB variable in the script

Let me know if you find this helpful or if you have comments, corrections or improvements.

Here is the script.

 #!/bin/bash # Initialize Variables DBCONFIG="--defaults-file=mysql-defaults.cnf" DB="aws_security_groups" SGLOOP=0 EC2LOOP=0 ELBLOOP=0 RDSLOOP=0 DEFAULTLOOP=0 OPSLOOP=0 CACHELOOP=0 DEL_GROUP="" # Function to report back # of rows function Rows { ROWS=`echo "select count(*) from groups" | mysql $DBCONFIG --skip-column-names $DB` # echo -e "Excluding $1 Security Groups.\nGroups Left to audit: "$ROWS echo -e $ROWS" groups left after Excluding $1 Security Groups." } # Empty the table echo -e "delete from groups where groupid is not null" | mysql $DBCONFIG $DB # Get all Security Groups aws ec2 describe-security-groups --query "SecurityGroups[*].[GroupId,GroupName,Description]" --output text > /tmp/security_group_audit.txt while IFS=$'\t' read -r -a myArray do if [ $SGLOOP -eq 0 ]; then VALUES="(\""${myArray[0]}"\",\""${myArray[1]}"\",\""${myArray[2]}"\")" else VALUES=$VALUES",(\""${myArray[0]}"\",\""${myArray[1]}"\",\""${myArray[2]}"\")" fi let SGLOOP="$SGLOOP + 1" done < /tmp/security_group_audit.txt echo -e "insert into groups (groupid, groupname, description) values $VALUES" | mysql $DBCONFIG $DB echo -e $SGLOOP" security groups total." # Exclude Security Groups assigned to Instances for groupId in `aws ec2 describe-instances --output json | jq -r ".Reservations[].Instances[].SecurityGroups[].GroupId" | sort | uniq` do if [ $EC2LOOP -eq 0 ]; then DEL_GROUP="'$groupId'" else DEL_GROUP=$DEL_GROUP",'$groupId'" fi let EC2LOOP="$EC2LOOP + 1" done echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB Rows "EC2 Instance" DEL_GROUP="" # Exclude groups assigned to Elastic Load Balancers for elbGroupId in `aws elb describe-load-balancers --output json | jq -c -r ".LoadBalancerDescriptions[].SecurityGroups" | tr -d "\"[]\"" | sort | uniq` do if [ $ELBLOOP -eq 0 ]; then DEL_GROUP="'$elbGroupId'" else DEL_GROUP=$DEL_GROUP",'$elbGroupId'" fi let ELBLOOP="$ELBLOOP + 1" done echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB Rows "Elastic Load Balancer" DEL_GROUP="" # Exclude groups assigned to RDS for RdsGroupId in `aws rds describe-db-instances --output json | jq -c -r ".DBInstances[].VpcSecurityGroups[].VpcSecurityGroupId" | sort | uniq` do if [ $RDSLOOP -eq 0 ]; then DEL_GROUP="'$RdsGroupId'" else DEL_GROUP=$DEL_GROUP",'$RdsGroupId'" fi let RDSLOOP="$RDSLOOP + 1" done echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB Rows "RDS Instances" DEL_GROUP="" # Exclude groups assigned to OpsWorks for OpsGroupId in `echo -e "select groupid from groups where groupname like \"AWS-OpsWorks%\"" | mysql $DBCONFIG $DB` do if [ $OPSLOOP -eq 0 ]; then DEL_GROUP="'$OpsGroupId'" else DEL_GROUP=$DEL_GROUP",'$OpsGroupId'" fi let OPSLOOP="$OPSLOOP + 1" done echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB Rows "OpsWorks" DEL_GROUP="" # Exclude default groups (can't be deleted) for DefaultGroupId in `echo -e "select groupid from groups where groupname like \"default%\"" | mysql $DBCONFIG $DB` do if [ $DEFAULTLOOP -eq 0 ]; then DEL_GROUP="'$DefaultGroupId'" else DEL_GROUP=$DEL_GROUP",'$DefaultGroupId'" fi let DEFAULTLOOP="$DEFAULTLOOP + 1" done echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB Rows "Default" DEL_GROUP="" # Exclude Elasticache groups for CacheGroupId in `aws elasticache describe-cache-clusters --output json | jq -r ".CacheClusters[].SecurityGroups[].SecurityGroupId" | sort | uniq` do if [ $CACHELOOP -eq 0 ]; then DEL_GROUP="'$CacheGroupId'" else DEL_GROUP=$DEL_GROUP",'$CacheGroupId'" fi let CACHELOOP="$CACHELOOP + 1" done echo -e "delete from groups where groupid in ($DEL_GROUP)" | mysql $DBCONFIG $DB Rows "ElastiCache" # Display Security Groups left to audit / delete echo "select * from groups order by groupid" | mysql $DBCONFIG $DB | sed 's/groupid\t/groupid\t\t/' 

And here is sql to create the database.

 -- MySQL dump 10.13 Distrib 5.5.41, for debian-linux-gnu (x86_64) -- -- Host: localhost Database: aws_security_groups -- ------------------------------------------------------ -- Server version 5.5.40-log /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; /*!40101 SET NAMES utf8 */; /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; /*!40103 SET TIME_ZONE='+00:00' */; /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; -- -- Table structure for table `groups` -- DROP TABLE IF EXISTS `groups`; /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE `groups` ( `groupid` varchar(12) DEFAULT NULL, `groupname` varchar(200) DEFAULT NULL, `description` varchar(200) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Dumping data for table `groups` -- LOCK TABLES `groups` WRITE; /*!40000 ALTER TABLE `groups` DISABLE KEYS */; /*!40000 ALTER TABLE `groups` ENABLE KEYS */; UNLOCK TABLES; /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; /*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */; /*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; /*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */; /*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */; /*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */; -- Dump completed on 2015-01-27 16:07:44 

Answer 5

Using the node.js AWS SDK, I can confirm that AWS does not allow you to remove the security groups that are used. I wrote a script that just tries to delete all groups and gracefully handles errors. This works for classic and modern VPCs. The error message can be seen below.

 Err { [DependencyViolation: resource sg-12345678 has a dependent object] message: 'resource sg-12345678 has a dependent object', code: 'DependencyViolation', time: Mon Dec 07 2015 12:12:43 GMT-0500 (EST), statusCode: 400, retryable: false, retryDelay: 30 } 

Answer 6

Boto example for printing group identifiers and names of only security groups that do not have current instances.

It also shows how to indicate which area you are in.

 import boto import boto.ec2 EC2_REGION='ap-southeast-2' ec2region = boto.ec2.get_region(EC2_REGION) ec2 = boto.connect_ec2(region=ec2region) sgs = ec2.get_all_security_groups() for sg in sgs: if len(sg.instances()) == 0: print ("{0}\t{1}".format(sg.id, sg.name)) 

To confirm which security groups are still in use, you must cancel or delete the test if len(sg.instances()) == 0 and print the value of len(sg.instances()) .

eg.

 print ("{0}\t{1}\t{2} instances".format(sg.id, sg.name, len(sg.instances()))) 

Answer 7

Among other features, ScoutSuite and Prowler report unused EC2 security groups. Both are open source.

Answer 8

For SGs connected to network interfaces:

By name:

 aws ec2 describe-network-interfaces --output text --query NetworkInterfaces[*].Groups[*].GroupName | tr -d '\r' | tr "\t" "\n" | sort | uniq 

By ID:

 aws ec2 describe-network-interfaces --output text --query NetworkInterfaces[*].Groups[*].GroupId | tr -d '\r' | tr "\t" "\n" | sort | uniq 

Answer 9

There is a tool on the AWS market that makes it a lot easier. It shows which groups are attached / detached for easy removal, but also compares your VPC flow protocols with the security group rules and shows which SG rules are used or not used. AWS posted an ELK stack solution for this, but it was ridiculously complicated.

Here is the tool and disclaimer I worked on. But I hope you find all this necessary: https://www.piasoftware.net/single-post/2018/04/24/VIDEO-Watch-as-we-clean-up-EC2-security-groups-in -just -few minutes

Answer 10

Unfortunately, the selected answer is not as accurate as I need (I tried to find out the reason, but I preferred to implement it).
If I check ALL NetworkInterfaces , looking for attachments in any SecurityGroup , it gets partial results. If I check only on EC2Instances , it also returns partial results to me.

So my approach to the problem:

1. I get ALL EC2 SecurityGroups - all_secgrp
2. I get ALL instances of EC2 - all_instances
3. For each instance, I connect all SecurityGroups to it
   1. I remove from all_secgrp each of these SecurityGroups (because I am attached)
4. For each SecurityGroup, I check the connection with any network interfaces (using the filter function and filtering using this security-group-id )
   1. If no connection is found, I am an all_secgrp security group from all_secgrp

In the application you can see a code snippet. Do not complain about efficiency, but try to optimize it if you want.

 all_secgrp = list(ec2_connector.security_groups.all()) all_instances = ec2_connector.instances.all() for single_instance in all_instances: instance_secgrp = ec2_connector.Instance(single_instance.id).security_groups for single_sec_grp in instance_secgrp: if ec2.SecurityGroup(id=single_sec_grp['GroupId']) in all_secgrp: all_secgrp.remove(ec2.SecurityGroup(id=single_sec_grp['GroupId'])) all_secgrp_detached_tmp = all_secgrp[:] for single_secgrp in all_secgrp_detached_tmp: try: print(single_secgrp.id) if len(list(ec2_connector.network_interfaces.filter(Filters=[{'Name': 'group-id', 'Values': [single_secgrp.id]}]))) > 0: all_secgrp.remove(single_secgrp) except Exception: all_secgrp.remove(single_secgrp) return all_secgrp_detached 

Answer 11

This is a complex problem if you have security groups that reference other security groups in the rules. If so, you will have to solve DependencyErrors, which is not trivial.

If you use only IP addresses, then this solution will work after creating the boto3 client:

 # pull all security groups from all vpcs in the given profile and region and save as a set all_sgs = {sg['GroupId'] for sg in client.describe_security_groups()['SecurityGroups']} # create a new set for all of the security groups that are currently in use in_use = set() # cycle through the ENIs and add all found security groups to the in_use set for eni in client.describe_network_interfaces()['NetworkInterfaces']: for group in eni['Groups']: in_use.add(group['GroupId']) unused_security_groups = all_sgs - in_use for security_group in unused_security_groups: try: response = client.delete_security_group(GroupId=security_group) except ClientError as e: if e.response['Error']['Code'] == 'DependencyViolation': print('EC2/Security Group Dependencies Exist') else: print('Unexpected error: {}'.format(e)) 

Answer 12

The easiest way to detect unused security groups in this VPC is to use aws-cli with the -stale-security-groups description option.

See the following documentation for more information.

https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-stale-security-groups.html

Hope this helps.