Lab 8 - Scripting (with DNS and Logs)

In this lab, you’re going to use scripting to automate some functions for Tiger Enterprises and gain a perspective on the contents of some key log files.

Activities

Part 1 - Automatic DNS Updates (with CLI and Scripting)

Currently, the load balancer and VPN instances have public IPs that change each time they boot. But what happens to their DNS record? www.STUDENT-NAME.tigerenterprises.org and vpn.STUDENT-NAME.tigerenterprises.org are still hard-wired to the original IP address when you first created them. Do you want to manually update that every time you start an instance?

Let’s script it so that the DNS record is automatically updated when the instance boots. And in the process, learn something about the Amazon AWS CLI.

References:

• https://dev.to/aws/amazon-route-53-how-to-automatically-update-ip-addresses-without-using-elastic-ips-h7o

First, create a Bash script that runs automatically when your EC2 instance starts. You'll need to do this twice, once for the load balancer instance, and again for the vpn instance. This script will:

• Set some variables for the hosted zone ID and desired domain name
• Retrieve the instance’s current public IP address
• Update the Route 53 record set with the new IP address

Be sure to replace DOMAIN-NAME with your chosen DNS subdomain (e.g. vpn.STUDENT-NAME.tigerenterprises.org), and replace HOSTED-ZONE-ID with your Route53 Hosted Zone as visible in the management console.

This script should be placed in the directory /var/lib/cloud/scripts/per-boot/ to run automatically at startup.

# Create the file.  Use sudo since destination is owned by root user
$ sudo nano /var/lib/cloud/scripts/per-boot/dns-update

The contents as the file should be as follows. Note this is a Bash script, which is the normal Ubuntu/Linux shell. You could type these commands one at a time in the terminal and get the same result. Also note that the .sh extension for shell scripts is purely optional. This script will work just fine with the name dns-update.

dns-update:

#!/bin/bash

# DNS settings for this instance (fixed)
DNS_ZONE=HOSTED-ZONE-ID
DNS_NAME=DOMAIN-NAME

# Runtime information about this instance (changes at boot!)
MY_IP=$(curl -s http://169.254.169.254/latest/meta-data/public-ipv4/)

# Update Route 53 Record Set based on the Name tag to the current Public IP address of the Instance
aws route53 change-resource-record-sets --hosted-zone-id $DNS_ZONE --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"'$DNS_NAME'","Type":"A","TTL":300,"ResourceRecords":[{"Value":"'$MY_IP'"}]}}]}' --profile classwide-account

Ensure that the script is marked as executable (otherwise it won’t run):

# Use sudo since file is owned by root user
$ sudo chmod +x /var/lib/cloud/scripts/per-boot/dns-update

Note that this script uses a special link-local network IP address 169.254.169.254 to retrieve instance metadata directly from the AWS management system. So rather than having to figure out how to obtain these details ourselves via convoluted parsing, we can query this information directly in a machine-readable format. Anything you can find in the AWS management console for your EC2 instance can be queried via this mechanism. Useful in many scenarios, not just this one!

Second, ensure the AWS CLI tool is installed. It’s not part of the Ubuntu base install.

References

• Install the AWS CLI version 2 on Linux

$ cd    # Run out of your home directory
$ sudo apt install unzip   # Unzip is not part of base install
# Download latest AWS CLI
$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
$ unzip awscliv2.zip
$ sudo ./aws/install

# Confirm that the AWS CLI is installed
$ aws --version
# Should see output like:
# aws-cli/2.2.40 Python/3.8.8 Linux/5.11.0-1017-aws exe/x86_64.ubuntu.20 prompt/off

At this point, the script is created and the CLI tools that the script depends on have been installed. What's left to do is provide authorization for the script to run. After all, it's changing data in the instructor AWS account via the aws route53 change-resource-record-sets command.

The AWS CLI credentials file is stored on a per-user basis in the location ~/.aws/credentials. Because your dns-update script will be run at startup as the root user, you need to specify the AWS credentials in the default location for the root user, and not for the "ubuntu" user you are currently logged in as.

$ sudo mkdir /root/.aws               # Create the settings directory for the root user
$ sudo nano /root/.aws/credentials    # Create the credentials file for the root user

In the credentials file, you need one section, named [classwide-account] that has the shared class credentials that are only used for Route53 updates. You can copy these values from the Canvas announcement. Notice that the bash script above contains the argument --profile classwide-account, which is how the aws route53 change-resource-record-sets command knows to use this specific authentication method.

[default]

[classwide-account]
aws_access_key_id=FILL THIS IN
aws_secret_access_key=FILL THIS IN

Once this is done, you should be good to go. You have a script that runs on boot, the AWS CLI installed, and have provided the class-wide credentials to run the aws route53 change-resource-record-sets command.

Time for a test! Reboot your load balancer and vpn instance with sudo reboot. Check in your Route53 console under the tigerenterprises.org hosted zone. Look at the Type "A" records for vpn. and www.. Are those IP addresses the instances' current public IPs? That means your script worked. If they are still set to the old public IP from last week, that means your script did not work. (See troubleshooting below). You can also try loading your website. Be sure to start the private web server first.

Troubleshooting:

Is the script not working? You can run it manually as a test via sudo /var/lib/cloud/scripts/per-boot/dns-update You should get some sort of machine-readable output indicating that your DNS update request has been accepted and is pending. Any errors should be investigated.

$ sudo su   # Become the ROOT user temporarily
$ /var/lib/cloud/scripts/per-boot/dns-update    # Run the script
{
    "ChangeInfo": {
        "Id": "/change/C05339001WLQFYDJOWIFF",
        "Status": "PENDING",
        "SubmittedAt": "2020-09-20T20:01:17.146000+00:00"
    }
}
$ exit  # Leave the root user

Part 2 - Log File Analysis

Let's examine some key log files of system events.

The log files for the NGINX web server are located at /var/log/nginx/

• Access log containing pages served: access.log (rotated to access.log.1, then compressed as access.log.2.gz, etc…)
  • You can customize the data logged, but the default “combined” file has the following fields and structure: '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" “$http_user_agent" (See nginx log format)
• Error logs containing errors encountered: error.log (rotated to error.log.1, then compressed as error.log.2.gz, etc…)

The log files for your computer logins (via SSH or interactively at the keyboard) are located in /var/log/auth.log (and rotated through auth.log.1, auth.log.2.gz, etc)

Here are some useful examples of using command line utilities to view or manipulate log files.

• View the entire access log file: cat /var/log/nginx/access.log
• View the contents of a gzip-compressed log file without uncompressing it first: zcat /var/log/nginx/access.log.2.gz
• View the log file one page at a time: less /var/log/nginx/access.log
• View the newest accesses (at the bottom of the file: tail /var/log/nginx/access.log
• View the oldest accesses (at the top of the file: head /var/log/nginx/access.log
• Follow the access file in real time as new pages are requested: tail -f /var/log/nginx/access.log
• Look for HTTP GET requests in the access file (but view it in a page-oriented fashion because the list may be long!): cat /var/log/nginx/access.log | grep "GET " | less
• Parse a file into “fields” that are “delimited” by a sequence of characters, and select which fields you want to examine: tail /var/log/nginx/access.log | cut -d " " -f 6,7
• And, of course, you recall file I/O redirection from previous discussions...

References:

• Configuring Logging for NGINX
• Bash Shell Scripting (wikibooks.org)
• Bash Scripting Tutorial
• Bash Programming Introduction How-To
• Advanced Bash-Scripting Guide

Script 1 - Parse NGINX Access Logs

Write a BASH script for Linux hosts that searches an NGINX access.log file for file requests that were answered with status=404 (File Not Found), and produces a list of file paths in the website (e.g. /page/file1.html) and a count of how often they occurred.

• The script should be called errorpages.sh
• The script should be executable: chmod +x errorpages.sh
• The first line of the script should contain the path to the interpreter: #!/bin/bash
• The script should accept one argument: The path/filename to search
• The script should combine existing command line utilities to accomplish this task. Your script should NOT launch a programming language (e.g. Python, Perl, etc..)
• The script should be able to search either an uncompressed log file (e.g. access.log) or a compressed log file produced by the log rotate tool (eg. access.log.10.gz). Do not delete the compressed log file! (Unless otherwise specified, extraction tools like gzip will removed the compressed archive.)
• The search should not leave a mess in the log directory. If you need to decompress a file to search it, put it in /tmp and clean up afterwards.
  • Tip: zcat might be a more elegant approach to viewing the contents of a compressed file
• The results should be in sorted order based on the frequency of occurrence. The most common 404 error file should be at the top of the list, labeled by the count of occurrences.
• For development purposes (because your NGINX access logs are presumably quite empty), a .zip archive of access.log files from the cyberlab web server is available for download and use.

Example output:

$ ./errorpages.sh /var/log/nginx/access.log.13.gz 
Examining file /var/log/nginx/access.log.13.gz
File is compressed archive - decompressing first
Decompressing to /tmp/access.log.13
     40 / 404
      2 /.env 404
      1 /.well-known/security.txt 404
      1 //2018/wp-includes/wlwmanifest.xml 404
      1 //2019/wp-includes/wlwmanifest.xml 404
      1 //blog/wp-includes/wlwmanifest.xml 404
      1 //cms/wp-includes/wlwmanifest.xml 404
      1 //media/wp-includes/wlwmanifest.xml 404
      1 //news/wp-includes/wlwmanifest.xml 404
      1 //shop/wp-includes/wlwmanifest.xml 404
      1 //site/wp-includes/wlwmanifest.xml 404
      1 //sito/wp-includes/wlwmanifest.xml 404
      1 //test/wp-includes/wlwmanifest.xml 404
      1 //web/wp-includes/wlwmanifest.xml 404
      1 //website/wp-includes/wlwmanifest.xml 404
      1 //wordpress/wp-includes/wlwmanifest.xml 404

Script 2 - Parse System Authentication Logs

Write a BASH script for Linux hosts that searches a system auth.log file for login attempts that were unsuccessful, and produces a list of timestamps, usernames and corresponding IP addresses sorted by the date the unsuccessful attempt was made (newest to oldest)

• The script should be called errorlogins.sh
• The script should be executable: chmod +x errorpages.sh
• The first line of the script should contain the path to the interpreter: #!/bin/bash
• The script should accept one argument: The path/filename to search
• The script should combine existing command line utilities to accomplish this task. Your script should NOT launch a programming language (e.g. Python, Perl, etc..)
• The script should be able to search either an uncompressed log file (e.g. auth.log) or a compressed log file produced by the log rotate tool (eg. auth.log.10.gz). Do not delete the compressed log file! (by default, extraction tools will removed the compressed archive)
• The search should not leave a mess in the log directory. If you need to decompress a file to search it, put it in /tmp and clean up afterwards.
  • Tip: zcat might be a more elegant approach to viewing the contents of a compressed file
• The results should be in reverse order based on the date of occurrence. The newest login attempts should be at the beginning with the oldest login attempts at the end.
• For development purposes (because your system auth logs are presumably quite empty), a .zip archive of auth.log files from the cyberlab web server is available for download and use.

Example Output:

ubuntu@cyberlab:~$ ./errorlogins.sh /var/log/auth.log.4.gz 
Examining file /var/log/auth.log.4.gz
File is compressed archive - decompressing first
Decompressing to /tmp/auth.log.4
Aug 31 06:23:26 admin 68.183.31.114
Aug 31 06:19:00 ks 186.23.211.154
Aug 31 06:10:30 liyan 180.76.173.75
Aug 31 06:08:15 uftp 201.149.55.53
Aug 31 06:05:53 pi 176.36.130.108
Aug 31 06:05:53 pi 176.36.130.108
Aug 31 06:05:18 ssl 134.122.112.119
Aug 31 05:25:01 r 106.12.183.209
Aug 31 05:18:09 joseph 152.136.230.126
Aug 31 04:39:26 postgres 188.195.182.146
Aug 31 04:32:15 ubnt 89.144.47.28

Part 3 - Limiting SSH Connections

If there are repeated failed attempts to log in from the same IP address, is that a legitimate user, or an attack? Should we do something about this?

Fail2ban is a tool that monitors system log files looking for IPs that show signs of malicious activity, such as repeated failed logon attempts. Then it updates the system’s internal firewall (not the AWS Security Group) to ban that IP address from taking further activity for a specified amount of time. Fail2ban can parse the log files for a wide variety of server applications, most notably OpenSSH (our SSH server) as well as the Apache and Nginx web servers.

References:

• Fail2Ban user manual

In Tiger Enterprises, currently the load balancer, VPN, and the Linux desktop are all Linux boxes reachable via SSH. Install Fail2Ban on all three systems to slow down any brute force attempts at password guessing.

Install Fail2ban:

$ sudo apt install fail2ban

# Or for the desktop instance which is Amazon Linux:
#  sudo yum install fail2ban 
#  sudo systemctl start fail2ban

Check which configurations are enabled. The best practice here to enable/disable features is not to modify the full configuration file (/etc/fail2ban/jail.conf, which is subject to being overwritten any time the package is updated). Instead, user customizations should be placed in a file in /etc/fail2ban/jail.d/. Look at the file there already:

cat /etc/fail2ban/jail.d/defaults-debian.conf

# Or for the desktop instance which is Amazon Linux:
#  sudo nano /etc/fail2ban/jail.conf
#  Add the line "enabled = true" inside the [sshd] section **halfway** through the file
#  (NOT the comment block up top)
#  Then restart the service:  sudo systemctl restart fail2ban
#  And verify it is running:  sudo systemctl status fail2ban

Verify that Fail2ban is active. By default, you should see the sshd process as in the “jail list”, indicating that logs for that service are being monitored.

$ sudo fail2ban-client status
#
# Expected output:
# Status
# |- Number of jail:    1
# `- Jail list: sshd

Have Fail2ban search the existing log files to see if any entries match the rules for a temporary ban

$ sudo fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf
# Look for a line:   Failregex: XXX total

Lab Deliverables

Upload to the Lab 8 Canvas assignment all the lab deliverables to demonstrate your work:

• Part 1 - Automated DNS Updates
  • Screenshot(s) showing the scripts used on the VPN and load balancer instances
  • Screenshot(s) showing your AWS credentials file with the classwide-account profile specified
  • Screenshot showing your website loading from https://www.STUDENT-NAME.tigerenterprises.org - It’s on the Honor Code that the DNS entry was automatically updated (by the script) and not manually updated by you
• Part 2 - Log File Analysis
  • Submit the two bash scripts (errorpages.sh and errorlogins.sh) to Canvas
• Part 3 - Limiting SSH Connections
  • Screenshot(s) showing fail2ban-client status on the load balancer, VPN, and Linux Desktop instances where the SSH server is being monitored (jailed).