Lab 7: Linux Network Security, Firewall, NAT Routing, and DHCP Server

Open an SSH remote terminal session to your Linux server’s IP address

Login with your standard user’s username and password

One common activity you would want to do when evaluating the security of a system is to find out what ports the system is accepting connections on. For this reason most operating systems have some kind of utility to display active network connections and open ports, Linux is no exception. Traditionally the netstat utility, which stands for network statistics, has been used to show open ports, incoming and outgoing connections, routing tables, interface statistics, masquerade connections, and multicast memberships. However, the more modern ss utility, which is short for socket statistics, can provide much of the same information in a much faster way, especially when there are a large number of connections on a server. Both tools can be used to identify unauthorized connections or traffic that could indicate a security breach, performance issues, or undesired applications running on the server. They are crucial for system and network administrators to ensure the integrity and efficiency of server operations.

Review the manual page for ss. Specifically, figure out what the -n -a -l -t -p and -u options do.

Try the ss -ntup and ss -tup commands and figure out what is being shown on your screen and in the output below. Be sure to run the commands with administrative permissions in order to have full access to see all the sockets open on your system and their details.

ben@2480-Z:~$ sudo ss -ntup
Netid              State              Recv-Q              Send-Q                           Local Address:Port                           Peer Address:Port               Process
tcp                ESTAB              0                   64                                172.17.50.36:22                             172.17.202.3:50358               users:(("sshd",pid=51036,fd=4),("sshd",pid=51020,fd=4))
ben@2480-Z:~$ sudo ss -tup
Netid              State              Recv-Q              Send-Q                           Local Address:Port                           Peer Address:Port               Process
tcp                ESTAB              0                   64                                172.17.50.36:ssh                            172.17.202.3:50358               users:(("sshd",pid=51036,fd=4),("sshd",pid=51020,fd=4))

Try the ss -nltup command and figure out what is being shown on your screen and in the output below:

ben@2480-Z:~$ sudo ss -nltup
Netid        State         Recv-Q        Send-Q                Local Address:Port                 Peer Address:Port        Process
udp          UNCONN        0             0                        127.0.0.54:53                        0.0.0.0:*            users:(("systemd-resolve",pid=334,fd=20))
udp          UNCONN        0             0                     127.0.0.53%lo:53                        0.0.0.0:*            users:(("systemd-resolve",pid=334,fd=18))
udp          UNCONN        0             0                           0.0.0.0:5355                      0.0.0.0:*            users:(("systemd-resolve",pid=334,fd=11))
udp          UNCONN        0             0                              [::]:5355                         [::]:*            users:(("systemd-resolve",pid=334,fd=13))
tcp          LISTEN        0             80                        127.0.0.1:3306                      0.0.0.0:*            users:(("mariadbd",pid=29657,fd=18))
tcp          LISTEN        0             4096                     127.0.0.54:53                        0.0.0.0:*            users:(("systemd-resolve",pid=334,fd=21))
tcp          LISTEN        0             128                         0.0.0.0:22                        0.0.0.0:*            users:(("sshd",pid=508,fd=3))
tcp          LISTEN        0             511                         0.0.0.0:80                        0.0.0.0:*            users:(("nginx",pid=48789,fd=5),("nginx",pid=48788,fd=5))
tcp          LISTEN        0             4096                        0.0.0.0:5355                      0.0.0.0:*            users:(("systemd-resolve",pid=334,fd=12))
tcp          LISTEN        0             4096                  127.0.0.53%lo:53                        0.0.0.0:*            users:(("systemd-resolve",pid=334,fd=19))
tcp          LISTEN        0             128                            [::]:22                           [::]:*            users:(("sshd",pid=508,fd=4))
tcp          LISTEN        0             511                            [::]:80                           [::]:*            users:(("nginx",pid=48789,fd=6),("nginx",pid=48788,fd=6))
tcp          LISTEN        0             4096                           [::]:5355                         [::]:*            users:(("systemd-resolve",pid=334,fd=14))

Notice that there are a number of ports open on your system. Some of these we have opened to provide a specific service such as SSH, Webserver, etc. but sometimes ports are open simply by default when some software has been installed. On older distributions of Linux there are even several ports open by default on fresh installs!

Note that in the local address column there are some addresses (such as the one for mariadb) listed as 127.0.0.x. These are services open only to the localhost (other software running on the server itself) and not to remote connections over the network. This is generally a more secure option if you don’t need to access the service remotely.

There are a number of different strategies you can use to secure your system including disabling a service, binding it to an internal-only IP address (such as a localhost address), or blocking access with a firewall rule. If your firewall is setup with an implicit (or explicit) reject any rule at the bottom of the input chain and you have not specifically opened a port it should not be accessible from other systems. We’ll explore the option of using a firewall to explicitly control access to ports on our system later in this lab.

Try the ss -natup command and figure out what is being shown on your screen and in the output below and how it differs from the previous command:

ben@2480-Z:~$ sudo ss -natup
Netid        State         Recv-Q        Send-Q               Local Address:Port                 Peer Address:Port         Process
udp          UNCONN        0             0                       127.0.0.54:53                        0.0.0.0:*             users:(("systemd-resolve",pid=334,fd=20))
udp          UNCONN        0             0                    127.0.0.53%lo:53                        0.0.0.0:*             users:(("systemd-resolve",pid=334,fd=18))
udp          UNCONN        0             0                          0.0.0.0:5355                      0.0.0.0:*             users:(("systemd-resolve",pid=334,fd=11))
udp          UNCONN        0             0                             [::]:5355                         [::]:*             users:(("systemd-resolve",pid=334,fd=13))
tcp          LISTEN        0             80                       127.0.0.1:3306                      0.0.0.0:*             users:(("mariadbd",pid=29657,fd=18))
tcp          LISTEN        0             4096                    127.0.0.54:53                        0.0.0.0:*             users:(("systemd-resolve",pid=334,fd=21))
tcp          LISTEN        0             128                        0.0.0.0:22                        0.0.0.0:*             users:(("sshd",pid=508,fd=3))
tcp          LISTEN        0             511                        0.0.0.0:80                        0.0.0.0:*             users:(("nginx",pid=48789,fd=5),("nginx",pid=48788,fd=5))
tcp          LISTEN        0             4096                       0.0.0.0:5355                      0.0.0.0:*             users:(("systemd-resolve",pid=334,fd=12))
tcp          LISTEN        0             4096                 127.0.0.53%lo:53                        0.0.0.0:*             users:(("systemd-resolve",pid=334,fd=19))
tcp          ESTAB         0             64                    172.17.50.36:22                   172.17.202.3:50358         users:(("sshd",pid=51036,fd=4),("sshd",pid=51020,fd=4))
tcp          LISTEN        0             128                           [::]:22                           [::]:*             users:(("sshd",pid=508,fd=4))
tcp          LISTEN        0             511                           [::]:80                           [::]:*             users:(("nginx",pid=48789,fd=6),("nginx",pid=48788,fd=6))
tcp          LISTEN        0             4096                          [::]:5355                         [::]:*             users:(("systemd-resolve",pid=334,fd=14))

Using the web browser on your administrative PC connect to your website at http://172.17.xx.xx/blog and then within a short period of time run the ss -natup and ss -ntup commands again. You should see something like this:

ben@2480-Z:~$ sudo ss -ntup
Netid        State        Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
tcp          ESTAB        0              0                       172.17.50.36:80                   172.17.202.3:55738         users:(("nginx",pid=48789,fd=12))
tcp          ESTAB        0              0                       172.17.50.36:80                   172.17.202.3:55737         users:(("nginx",pid=48789,fd=11))
tcp          ESTAB        0              64                      172.17.50.36:22                   172.17.202.3:50358         users:(("sshd",pid=51036,fd=4),("sshd",pid=51020,fd=4))
tcp          ESTAB        0              0                       172.17.50.36:80                   172.17.202.3:55733         users:(("nginx",pid=48789,fd=4))

Notice how you can see now the open TCP connections between Nginx and the web browser on your administrative PC as well as the IP address of your administrative PC. This can be a useful tool for seeing what systems are connected to your server, on what port they are connected, and what software or process on the server they are connected to.

Sometimes we want to verify what ports are open on a system by checking from another computer. The nmap, or Network Mapper, utility is a very powerful security scanning utility available on Linux. While ss or netstat use information from the Linux kernel about what ports and connections are in use by what processes nmap actively probes and tests ports on your system or another system to determine whether the port is open or not as well as additional information about the port in some cases. nmap is a complex and powerful utility. Entire books and extensive documentation are available which you may want to reference but we’ll only be exploring some of the more basic features in this introductory lab activity.

nmap provides a system on the Internet which they allow you to scan for testing purposes so let’s try a verbose scan which gives additional diagnostic detail by running the nmap -v scanme.nmap.org command which will take a minute to run:

ben@2480-Z:~$ nmap -v scanme.nmap.org
Starting Nmap 7.93 ( https://nmap.org ) at 2024-06-25 12:03 CDT
Initiating Ping Scan at 12:03
Scanning scanme.nmap.org (45.33.32.156) [2 ports]
Completed Ping Scan at 12:03, 0.05s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 12:03
Completed Parallel DNS resolution of 1 host. at 12:04, 13.00s elapsed
Initiating Connect Scan at 12:04
Scanning scanme.nmap.org (45.33.32.156) [1000 ports]
Discovered open port 80/tcp on 45.33.32.156
Discovered open port 22/tcp on 45.33.32.156
Discovered open port 31337/tcp on 45.33.32.156
Discovered open port 9929/tcp on 45.33.32.156
Completed Connect Scan at 12:04, 0.73s elapsed (1000 total ports)
Nmap scan report for scanme.nmap.org (45.33.32.156)
Host is up (0.048s latency).
Other addresses for scanme.nmap.org (not scanned): 2600:3c01::f03c:91ff:fe18:bb2f
Not shown: 996 closed tcp ports (conn-refused)
PORT      STATE SERVICE
22/tcp    open  ssh
80/tcp    open  http
9929/tcp  open  nping-echo
31337/tcp open  Elite

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 17.82 seconds

You can see in the output above that nmap discovered four open ports on the scanme.nmap.org system at the time of this scan. Furthermore, nmap was able to determine the likely service running on each of those ports and show the name of the service.

The -v option makes the output more verbose which means more detail is provided. Try again, but without the -v option this time to get just the standard output. Use the command nmap scanme.nmap.org and take note of what the difference is while the command is running.

When scanning your own system there are a few different ways to go about it. You could either scan the localhost address 127.0.0.1 or the actual outside IP address of your system. You could also setup a separate system or VM and do the scanning from that system. In each case you might see somewhat different results, can you guess why?

The answer is related to how you have firewall rules setup and what addresses you have services bound to. For example by default on Debian systems the MySQL/MariaDB server daemon only listens for connections on the localhost address (127.0.0.1) and not on outside interfaces.

Try running the nmap 127.0.0.1 command and then compare output with the nmap <your outside 172.17.xx.xx ip address here> command. Do you see some network services listening only on the localhost address? These services are not accessible from outside your computer even though the ports are open and you would see them as open with ss.

The most realistic use of nmap though is to scan like an attacker would using a system outside of the one you’re testing. In fact it’s even possible to scan a whole range of IP addresses at once. Use nmap to scan the entire Linux System Administration class subnet with the nmap 172.17.50.0/23 command. Note that this could take some time depending on how many systems respond to nmap.

Take a look at the nmap book or documentation and try running a few different types of nmap scans on one of the systems you found in the Linux System Administration class subnet.

nmap also supports scanning IPv6 addresses but it needs to be done separately from IPv4. Note that a running service is not necessarily listening on both IPv4 and IPv6 addresses just because you have them both active on your operating system. Figure out how to scan IPv6 addresses with nmap and try scanning both an IPv4 and IPv6 address of your machine and compare the results. Use the same type of address (i.e. both IPv4 and IPv6 addresses should be the localhost addresses or should both be outside addresses) Are the same services open on both IPv4 and IPv6 on your system?

One of the important steps towards securing any system is to limit the attack surface. When we’re talking about a networked system one way that this is typically done is to make certain that only those ports we want to have open to the world are actually open. One of the most common ways of doing this is to use a host-based firewall running on the system. Like all modern network operating systems Linux has a built-in host-based firewall. Tradtionally the Linux firewall was based around iptables but starting in 2014 distributions began replacing that with the newer and more flexible nftables solution.

While it was somewhat common to write your own firewall rules directly in the iptables era the additional flexibility provided by nftables has made writing rules more complex. Many ditributions, including Debian, still provide compatible versions of the iptables commands which actually create nftables rules under the hood but it’s also become increasingly more common to use a utility to manage the firewall rules instead of writing them directly yourself. One reason for this is to reduce the complexity of managing a firewall by automatically handling things like implementing the rules automatically each time the system boots.

Two of the most common tools for managing firewall rules on Linux systems are firewalld and ufw. While ufw provides a basic host-based firewall it still relies on iptables or at least iptables compatible commands to implement it’s rules. On the other hand firewalld supports nftables natively and also supports defining zones and multiple complex rulesets such as those needed when multiple interfaces are connected to different networks. In this lab activity we’ll explore the use of firewalld to manage the nftables firewall on our server.

Begin by installing the firewalld package on your system.

Use the firewall-cmd --state command to check and see if the firewall is now up and running

ben@2480-Z:~$ sudo firewall-cmd --state
running

Check to see what rules are currently applied in your firewall using the firewall-cmd --list-all command:

ben@2480-Z:~$ sudo firewall-cmd --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces:
  sources:
  services: dhcpv6-client ssh
  ports:
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

By default all interfaces are in the public zone (this is set in the /etc/firewalld/firewalld.conf file). We can create custom zones or use pre-defined security zones which will speed up configuring the firewall. Take a minute to read more about the pre-defined zones in firewalld documentation.

Before we start working on rules for our firewall we should assign (move) our outside network interface (ens192) to the external zone which is designed for NAT routing which we’ll be setting up shortly. Do this by running the firewall-cmd --zone=external --add-interface=ens192 command.

Check to see what rules are enabled for the external zone. This time you will need to specify the zone name as it’s not the default public zone anymore. Use the firewall-cmd --zone=external --list-all command to do this.

You can also check the rules on all zones with the firewall-cmd --list-all-zones command.

If you were watching carefully you may have noted that the dhcpv6-client service was allowed in the public zone is not allowed by default on the external zone. This will prevent our server from being able to receive a DHCPv6 IP address on our ens192 interface anymore. Since we want to allow that run the firewall-cmd --zone=external --add-service="dhcpv6-client" command to allow the dhcpv6-client service on interfaces in the external zone.

Try accessing your website again from your administrative PC by browsing to http://172.17.xx.xx/blog. You should notice that your site is no longer accessible because your firewall is blocking traffic on port 80 from getting to nginx!

Run the firewall-cmd --get-services command on your system to see what services firewalld can create rules for.

Try creating a new rule on the external zone to allow access to the http service from that zone. If you have done this successfully your external zone rules should now look like the ones below and your site should be accessible again from the administrative PC browser.

ben@2480-Z:~$ sudo firewall-cmd --zone=external --list-all
external (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources:
  services: dhcpv6-client http ssh
  ports:
  protocols:
  forward: yes
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

There is one more very important thing to know about firewalld. By default, changes you make to the firewall are applied instantly. However, when the system is rebooted everything will go back to the same configuration that was in place the last time the system booted. This means if you make an error when configuring the firewall and lock yourself out of the system a reboot of the system will most likely get you back in. However, once you have tested and verified that your rules are working the way you want you need an extra command to save them permanently to be applied the next time the system boots. Run the firewall-cmd --runtime-to-permanent command now to save automatically apply your current firewall rules the next time the system boots.

One of the things you may want to do with a Linux server is to use it as a router. To do this we’ll need to use a second network interface card attached to a second network. Our virtual lab environment is already setup with a second network interface card (ens224) in our Linux server which is attached to a small network with just one client computer. While it’s possible to install routing protocol software (such as OSPF or BGP) in Linux and use it as a regular enterprise router it’s probably more likely that you’ll want to use it as a SOHO NAT router. In fact, many of the commercial SOHO NAT routers use Linux themselves!

The first thing we need to do is to configure that second NIC (ens224) with an IP address. Create a new /etc/systemd/network/20-internal.network file just like we did back in lab 5 for our primary network.

Apply the changes to our network configuration run the systemctl restart systemd-networkd command as the admninistrative user.

Check that the ip_forward permission is enabled on your Linux server by running sysctl net.ipv4.ip_forward as thge administrative user and making sure that the output shows a 1 for enabled.

Put the second NIC (ens224) into the internal zone of firewalld

Modify the firewall to allow access to the ssh and http services on your Linux server from clients on the internal zone.

Ensure that all devices on your internal zone will share one IP on your external zone. This is commonly called NAT but is really called port address translation (PAT) by some vendors such as Cisco and NAT masquerading by Linux. This is actually a setting on the external zone and not the internal zone. You should see a masquerade: yes line in the rules for the zone. If you don’t or need to enable it on a custom outside zone you can do so with a command like firewall-cmd --add-masquerade --zone=external.

Add a policy to your firewall which allows traffic from the internal zone to the external zone

Just as before once you are satisfied your firewall is running correctly you should use the firewall-cmd --runtime-to-permanent command to set these rules to automatically load each time the system is started.

Reload the firewall with the firewall-cmd --reload command.

We want our inside client network to receive IPv4 addresses automatically from a DHCP server so we’ll configure our Linux Server to also act as a DHCPv4 server using the ISC Kea DHCP server software.

Begin by installing the kea-dhcp4-server package on your server.

There are a lot of options that can be used for an advanced DHCP server. The configuration file for Kea is explained in the Kea Administrator Reference Manual but for our purposes of a very simple DHCP server we don’t need too much configuration. Let’s move the existing sample configuration file from /etc/kea/kea-dhcp4.conf to /etc/kea/kea-dhcp4.bak to keep it as a backup just in case we want to reference it.

After moving the sample Kea configurtion file to a backup copy start nano as an adminsitrative user with a new blank /etc/kea/kea-dhcp4.conf file and put the following contents into it and save the file:

{
    "Dhcp4": {
        "interfaces-config": {
            "interfaces": [
                "ens224"
            ]
        },
        "lease-database": {
            "type": "memfile",
            "persist": true,
            "name": "/var/lib/kea/kea-leases4.csv",
            "lfc-interval": 3600
        },
        "valid-lifetime": 3600,
        "option-data": [
            {
                "name": "domain-name-servers",
                "data": "172.17.50.1"
            }
        ],
        "subnet4": [
            {
                "id": 1,
				"subnet": "192.168.1.0/24",
                "pools": [
                    {
                        "pool": "192.168.1.100 - 192.168.1.254"
                    }
                ],
                "option-data": [
                    {
                        "name": "routers",
                        "data": "192.168.1.1"
                    }
                ]
            }
        ]
    }
}

Kea uses JSON style configuration files. These (and the somewhat related YAML format) are becoming a popular style of modern software configuration file. In this case we have specified:

Use the systemctl restart kea-dhcp4-server command to restart the server with the new configuration and systemctl status kea-dhcp4-server to make sure the DHCP server has restarted properly.

Make sure to add a firewall rule allowing clients on the internal zone to access the dhcp service and to set the new rule to load automatically each time the system is started. Otherwise the firewall will block systems in the internal zone from contacting the DHCP server to get an address! We’ll need to add a new firewall rule like this (plus probably one for the external zone too) every time we add a new service to our server from this point on!

Because we are going to use a GUI installation of Linux to serve as an example client PC behind our Linux Server we’ll need to begin by installing a copy of Linux on that client PC system. We’ll use a differnt distribution of Linux, Linux Mint, for this system. Although it’s possible to install a GUI on Debian, Linux Mint has been created specifically as an easy to use client version on Linux. It’s still based on Debian so much of what we have learned so far will still apply as we’ll see.

Because we want to use the local console of the virtual machine for the installation and use of Linux Mint we’ll need to log in to the ITC NetLab system and access a reservation for your pod.

Access your virtual machine console

Install Linux Mint using the graphical wizard

Complete the installation

When your Linux Mint system reboots log in with the username and password established during the installation process.

You should see a welcome screen the first time you log in. You can uncheck the "Show this dialog at startup" box in the lower right corner and then close the dialog with the "X" in the upper right corner.

Adjust your Linux Mint display resolution to be able to see more content on the screen.

Check the IP address and test network connectivity

Use the Linux Mint Software Manager to Install an SSH server on the Mint system

Spend a few more minutes exploring the functionality of the Linux GUI and desktop with Linux Mint.

Now we are going to enable port forwarding through our Linux Server to the Linux Desktop VM. This will allow you to access the SSH server you setup on the Linux Mint system from the ITC network and VPN or Administrative PC without needing to SSH to the Linux Server first. Because we are already using port 22 on our Linux server’s outside interface to provide SSH access to our Linux server itself we need to use a different port to access the Linux Mint desktop system. Instead of changing the port number in the SSH server software on Linux Mint we can have our firewall do it at the same time the NAT masquerading is occurring.

On your Linux server use the command firewall-cmd --add-forward-port=port=2222:proto=tcp:toport=22:toaddr=192.168.1.100 --zone=external where 192.168.1.100 is the IP address of your Linux Mint system to create a new rule which forwards port 2222 on the IP address of your Linux server through to port 22 on the IP address of your Linux Mint system.

From a computer on the ITC network, such as one connected to the VPN or the Administrative PC, try to SSH to port 2222 on the outside (172.17.50.xx) IP address of your Linux server system. Use a username and password from your Linux Mint desktop system. Upon connection you should see that you are actually connected to the Linux Mint system via port forwarding.

Once you are satisfied your firewall is running correctly use the correct firewall-cmd command to save the current runtime rules as permanent rules.

When you are done testing this section of the lab please power off your new Linux Mint VM using the shutdown command in the Mint Menu as the desktop systems require far more RAM and CPU power to run than the Linux Server VMs.

If you have port 22 exposed to the Internet (remember, our lab servers are not directly exposed to public IP addresses and need to be accessed via VPN or Netlab) and watch your system logs you will see a steady stream of attackers trying to brute force their way into your Linux server by using automated tools to guess usernames and passwords. While using strong passwords can help there are some better tools at our disposal. One of these tools is the SSH Key system where the client uses a secret key file to authenticate to the SSH server instead of (or in addition to) a password. So long as the secret key file is not available to attackers they will not be able to log in to the system. For this reason some administrators enforce a rule that passwords cannot be used alone for remote SSH access and all remote SSH authentication must be by SSH key.

SSH Keys are created by the client software and not by the server so the exact method used to create them depends on the client software you are using to connect. If you are using PuTTY as your SSH client on either your own system or on the Administrative PC follow the instructions below to implement SSH keys. If you are using a command line SSH client (for example you are connecting from a Mac or a Linux computer of your own) skip this sub-section and go to #3 where you’ll find instructions for that.

SSH Keys are created by the client software and not by the server so the exact method used to create them depends on the client software you are using to connect. If you are using a command line OpenSSH client (for example you are connecting from a Mac or a Linux computer of your own) follow the instructions below to implement SSH keys. If you are using PuTTY as your SSH client on either your own system or on the Administrative PC you should have followed the instructions in the above sub-section #2 and skip this sub-section, pick up again at #4.

Note that your Linux server is currently configured to allow you to log in using either a username and password or SSH key based authentication. If you wanted to require users use SSH key based authentication and not allow passwords (which prevents the types of brute force password attacks mentioned at the beginning of this section) you would need to edit the OpenSSH options on your Linux server in the /etc/ssh/sshd_config file to activate the PasswordAuthentication no line and then restart the ssh server. Do not do this on your Linux server for this class as we will be doing some more work with password based SSH authentication.

An alternative way to secure SSH from brute force password attacks is to use multi-factor authentication. In this section we’ll setup a time-based one time password (TOTP) second fadctor of authentication (2FA) which will require a changing numeric code be entered along with your password when logging in. You will need to install a TOTP app on your phone or other device to show you the correct code to enter. The lab is written using the "Google Authenticator" phone app but there are many other TOTP apps which are compatible as well including options you can install on your Windows PC or Mac if you don’t have or want to install a phone app. However, you’ll be on your own as to how to add the required key into those apps if you choose to use something other than Google Authenticator.

Use apt to install the libpam-google-authenticator package on your Linux server system.

Run the google-authenticator command to create a new secret key in your home directory.

Install the "Google Authenticator" app on your phone (or whatever other TOTP app you would prefer on your phone or PC/Mac) and open it.

Add a code or enter the secret key in your TOTP authentication app. If you’re using Google Authenticator the easiest way to do this is tell it you want to scan a QR code and then point the camera on your phone at the QR code in the SSH window from your Linux server.

When things are setup in your TOTP app you should see a changing six digit code on your TOTP app. You will need to enter in the code on your Linux server in the SSH window where it says "Enter code from app". This will confirm that your phone app and server are both in sync.

Assuming the code entered was successful you should get a message "Code confirmed" as well as some emergency scratch codes you could save somewhere safe in case of an emergency which will each work one time to allow access.

You will be promted to answer four more questions. Read through each question and answer "y" to the first three and "n" to the last one (we’ll deal with rate limiting separately in the next part of the lab).

Do you want me to update your "/home/ben/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) n

Use administrative permissions to open the the /etc/ssh/sshd_config file on your system with a text editor.

Find the two settings below in the file and ensure both are set to yes then save the file and exist the editor.

UsePAM yes
ChallengeResponseAuthentication yes
KbdInteractiveAuthentication yes

Use administrative permissions to open the the /etc/pam.d/sshd file on your system with a text editor.

Just below the @include common-auth line towards the top of the file add these two lines

# two-factor authentication via Google Authenticator
auth   required   pam_google_authenticator.so

Use the systemctl restart ssh command to restart the SSH server.

Try making a new SSH connection to the server. You should be prompted for a password and then a verification code when connecting. Get the current verification code from your TOTP app.

Make sure you cannot log in with an incorrect verification code or without a verification code.

You can leave verification codes enabled on your Linux server for the rest of the course if you would like but you may want to turn them back off. To do this just open the /etc/pam.d/sshd file on your system and put a # at the beginning of the auth required pam_google_authenticator.so line like:

# two-factor authentication via Google Authenticator
#auth   required   pam_google_authenticator.so

Save and close the /etc/pam.d/sshd file and the verification codes should be turned off.

Yet another way to resist brute force password attacks against your Linux server is to monitor failed login attempts and block IP addresses in the system firewall for some amount of time if they have attempted to gain access multiple times and failed. This type of monitoring and blocking can be automated with software, one of the most popular software packages to do this with on Linux is called Fail2Ban because it bans the ip addresses (usually for some amount of time) if they fail the login process too many times. While the most common us is probably to monitor SSH remote login attempts the Fail2Ban software can be adjusted to monitor many other programs which log failed login attempts as well. It is not a bad idea to implement monitoring and blocking software like this even if your system is protected by other measures such as SSH keysm, 2FA, or required VPN access.

Use the apt utility to install the fail2ban package on your system.

Use a text editor to open the /etc/fail2ban/jail.conf file and read through the commented notes in the file. This file contains many of the configuration settings for the fail2ban application. Note that the file instructs you not to modify /etc/fail2ban/jail.conf but instead to create a new /etc/fail2ban/jail.local file with any modifications. Doing it this way will allow apt to update the defaults in /etc/fail2ban/jail.conf as new versions are released while still allowing you to customize settings as needed. After you have reviewed the commented notes in the /etc/fail2ban/jail.conf file close the file.

Use a text editor (with administrative permissions) to create and edit a new file /etc/fail2ban/jail.local where we’ll store our customizations. Copy and paste the customizations below into the file and save it. Note the comment lines below which also explain what each configuration line in the file is doing, the comments do not need to be copied but it should be OK to copy them too if you want.

# Change some default settings which affect all applications
# being monitored by the fail2ban application
[DEFAULT]

# Our Debian system is using the newer nftables for it's
# underlying firewall instead of the older iptables so we
# want to have fail2ban use native nftables commands for
# banning IP addresses which misbehave. There are two
# different banactions which need to know what backend
# to use banaction and banaction_allports so we'll set
# them both to use nftables.
banaction = nftables-multiport
banaction_allports = nftables-allports

# Change settings specifically around monitoring of
# SSH authentication attempts.
[sshd]

# Our version of Debian has switched to using systemd for
# SSH logging so we need to let fail2ban know to use the
# systemd backend or it won't be able to find an SSH log to
# monitor for failed authentication attempts.
backend=systemd

# Tell fail2ban it should monitor SSH attempts
enabled = true

# Tell fail2ban that if someone tries to break in to SSH
# they should be blocked from connecting to the server on
# ANY port, be default they would still be able to access
# ports other than the SSH port (22) such as our website.
# This will block them from getting to anything on the system.
port = 0:65535

# Tell fail2ban to block attackers for 1 minute. Normally
# you would probably want this to be longer, like 10 or 15
# minutes. But because we're pretty secure behind our VPN
# and we want to be able to do some testing we'll set it to
# cancel the ban after 1 minute.
bantime = 1m

# Tell fail2ban to block people who give a wrong login/password
# for SSH 3 times within 2 minutes.
findtime = 2m
maxretry = 3

Now that we have changed the configuration we should restart fail2ban with the systemctl restart fail2ban command. Always check to make sure the service has restarted and is running with the systemctl status fail2ban command.

Before testing out fail2ban for real you may want to simulate a ban on an IP address that is not being used just to see the firewall rules are being properly entered.

Now let’s do an actual intrusion test to see fail2ban in action.

Think about what you could do to access your Linux Server in case you accidentally got banned yourself? How could you get back in before the ban expired? If you got back in is there a command for fail2ban you can find which will let you remove the ban on a certain IP address?

Properly shutdown your Linux Mint Desktop system either using the menu in the lower left of the GUI or using an SSH or terminal session to the desktop system and the shutdown -h now command with administrator permissions.

Close the SSH session

If you are using the Administrative PC in Netlab instead of your own computer as the administrative computer you should also shut down that system in the usual way each time you are done with the Netlab system and then end your Netlab Reservation. You should do these steps each time you finish using the adminsitrative PC in future labs as well.

End your Netlab Reservation