AWS: Wherein I do things the hardest way possible.

I've used AWS for various projects, but not gotten extremely in depth. I've recently taken a contract that has stretched my AWS skills to the limit, and I've had to learn lots. I'm going to catalog some things senarios that others might find useful.

Give me all external Interface IPs so I can scan them

So, I need to lock down an environment. I need all the IP addresses. There's likely tons of ways to do this, here's how I torturted myself.

Option 1

aws ec2 describe-network-interfaces \
  --query "NetworkInterfaces[].Association.PublicIp" \
  --output yaml | sort -V | awk '{print $2}'

This lists ALL the network interfaces in your AWS account. This might be perferable depending on your needs.

So, you can list your interfaces and output them in json, yaml, text, and table. Now I would've thought that table would give me an IP address per line, it doesn't it just gloms then all on the same line, probably with a tab delimiter or something, I was so disgusted with this output I switched to YAML, since it required little processing.

Alternatively I could've used jq to process the default json output. jq isn't always installed everywhere, so I opted for regularly installed tools. sort -V properly sorts IP addresses, and awk removes the - from the yaml output.

MacOS users, consider using all GNU cli utilies instead of apple utilies. awk isn't different, but sort is.

❯ sort --version
2.3-Apple (106)
❯ gsort --version
sort (GNU coreutils) 8.32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

I don't know what apple sort is, however gsort is posix compliant and the arguments you learn with it will translate to linux, or windows cygwin/WSL.

brew install coreutils

THEN.. I discovered why it was putting it all on one line, if you don't specify the final value as anything other than a string, it will just output the value with nothing else, always surround the final value in brackets, and you'll get the one line per functionality you'd assume is default.

aws ec2 describe-network-interfaces \
  --query "NetworkInterfaces[].Association.[PublicIp]" \
  --output=text

Option 2

aws ec2 describe-instances \
  --query "Reservations[*].Instances[*].[PublicIpAddress]" \
  --output=text | sort -V

This will list only IP addresses from your instances. This won't list all public IP addresses from other services you might have with AWS.

Oddly, again, the output threw me. I was getting an annoying None in the output. Ugh.. Fine. I'll grep -v None it.

aws ec2 describe-instances \
  --query "Reservations[*].Instances[*].[PublicIpAddress]" \
  --output=text | sort -V | grep -v None

but... why?

Turns out in this output, the PublicIPAddress is handled differently in the raw data that comes back from AWS. Specifically calling it out in [PublicIPAddress] creates a condition where you're WANTING null values to be output. The fix for this was removing the []

aws ec2 describe-instances \
  --query "Reservations[*].Instances[*].PublicIpAddress" \
  --output=text | sort -V

Boom. Output is 1 address per line, suitable for a text file I can feed to nmap.

NMAP ... or shodan?!

nmap -p- -sT -T4 -vvvv -Pn --open -iL scanip -oA scanmeip

So.. you wanna scan a range? Start with nmap, or zenmap. If you aren't sure what all this is, ExplainShell might be able to help you, it's a great site, but doesn't stay up on all the arguments.

-p- is the same as -p 1-65535 Meaning scan every port possible.

-sT is perform a standard TCP port check, no fancy syn stuff, we're just looking to find out what ports are open.

-T4 this is Timing, 1 being nearly benign, and 5 might actually miss open ports because it'll tax your network connection.

-vvvvvvvvvvvvvvvvvvvvv veeerrrbbbooosssiitttyyyyyy

-Pn don't ping, assume it's alive

--open report ONLY on open ports

-iL scanme get the target list from the file scanipwhich I created from theaws ec2 describe-network-interfaces` above.

-oA scanmeip will create 3 files, a standard report, a greppable output, and most importantly an XML file you can transform into an HTML report!

Once you're done. Clone this repo: https://github.com/honze-net/nmap-bootstrap-xsl and ensure you have an xml processor installed (macOS comes with a workable xsltproc binary).


xsltproc -o scanme.html ./nmap-bootstrap-xsl/nmap-bootstrap.xsl scanme.xml


Next thing you know you have a beautifully formatted report showing all visible reports of all your externally visible IP addresses!

Then you get to cry a little at how much work you have to secure everything because the contract you walked into is way larger than you expected and no one ever set anything up correctly and you wonder why database ports are exposed to the public and someone took the time to install fail2ban but never enable or configure it and all the ssh listeners are configured for aws but everyone uses ubuntu login and the list goes on and on.. Life of ops nerds, lol.

Shodan?

Bonus points for anyone using shodan.io to scan your external IP addresses using the cli. If anyone is interested, I'll post how I did it.

wait.. there's points?