Lab 12: Filesystem Management and Hardware Utilities

Open an SSH remote terminal session to your Linux server’s IP address

Login with your standard user’s username and password

The Linux kernel is the program or code at the core of the operating system which essentially sits between the applications running on the system and the hardware of the system. It is ultimately responsible for facilitating interactions between the applications and hardware including the CPU, memory, and other system devices as well as preventing conflicts applications on the system. One of the key features of an operating system and reason for having one is that it allows applications to function at a higher level, they don’t need to think about how to communicate with all the different network cards on the market or all the different video cards. Instead the application can just tell the operating system to "send out this data on the network" or "display this data on the screen" and the operating system figures out how to do that with the hardware in the system. In order for the operating system to be able to communicate itself with all the different hardware you have in a system it normally uses device drivers which need to be somehow connected to the kernel. In Linux this connection is usually done with kernel modules which allow code (usually device drivers, filesystem drivers, network protocol drivers, and system calls) to be added to the kernel without rebuilding the kernel or even without restarting the system.

View a list of kernel modules which are loaded on your system using the lsmod command as an administrator which will show you the name of the module, how much memory it is using, and which other modules are relying on it:

ben@2480-Z:~$ sudo lsmod
Module                  Size  Used by
xt_nat                 16384  0
xt_tcpudp              20480  0
veth                   36864  0
xt_conntrack           16384  1
xt_MASQUERADE          20480  1
nf_conntrack_netlink    57344  0
xfrm_user              53248  1
xfrm_algo              16384  1 xfrm_user
xt_addrtype            16384  2
nft_compat             20480  4
br_netfilter           36864  0
bridge                311296  1 br_netfilter
stp                    16384  1 bridge
llc                    16384  2 bridge,stp
overlay               163840  0
nft_objref             16384  1
nf_conntrack_netbios_ns    16384  1
nf_conntrack_broadcast    16384  1 nf_conntrack_netbios_ns
nft_masq               16384  1
nft_fib_inet           16384  1
nft_fib_ipv4           16384  1 nft_fib_inet
nft_fib_ipv6           16384  1 nft_fib_inet
nft_fib                16384  3 nft_fib_ipv6,nft_fib_ipv4,nft_fib_inet
nft_reject_inet        16384  10
nf_reject_ipv4         16384  1 nft_reject_inet
nf_reject_ipv6         20480  1 nft_reject_inet
nft_reject             16384  1 nft_reject_inet
nft_ct                 24576  8
nft_chain_nat          16384  6
nf_nat                 57344  4 xt_nat,nft_masq,nft_chain_nat,xt_MASQUERADE
nf_conntrack          188416  9 xt_conntrack,nf_nat,nft_ct,nf_conntrack_netbios_ns,xt_nat,nf_conntrack_broadcast,nf_conntrack_netlink,nft_masq,xt_MASQUERADE
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
ip_set                 61440  0
cfg80211             1142784  0
rfkill                 36864  2 cfg80211
nf_tables             303104  541 nft_ct,nft_compat,nft_reject_inet,nft_fib_ipv6,nft_objref,nft_fib_ipv4,nft_masq,nft_chain_nat,nft_reject,nft_fib,nft_fib_inet
libcrc32c              16384  3 nf_conntrack,nf_nat,nf_tables
nfnetlink              20480  6 nft_compat,nf_conntrack_netlink,nf_tables,ip_set
intel_rapl_msr         20480  0
intel_rapl_common      32768  1 intel_rapl_msr
intel_uncore_frequency_common    16384  0
ghash_clmulni_intel    16384  0
sha512_ssse3           49152  0
sha512_generic         16384  1 sha512_ssse3
sha256_ssse3           32768  0
sha1_ssse3             32768  0
vsock_loopback         16384  0
vmw_vsock_virtio_transport_common    53248  1 vsock_loopback
vmw_vsock_vmci_transport    36864  1
vsock                  53248  5 vmw_vsock_virtio_transport_common,vsock_loopback,vmw_vsock_vmci_transport
vmwgfx                372736  1
aesni_intel           393216  0
crypto_simd            16384  1 aesni_intel
cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
drm_ttm_helper         16384  1 vmwgfx
rapl                   20480  0
ttm                    94208  2 vmwgfx,drm_ttm_helper
vmw_balloon            24576  0
pcspkr                 16384  0
drm_kms_helper        212992  3 vmwgfx
vmw_vmci               98304  2 vmw_balloon,vmw_vsock_vmci_transport
ac                     20480  0
button                 24576  0
evdev                  28672  2
joydev                 28672  0
serio_raw              20480  0
sg                     40960  0
binfmt_misc            24576  1
drm                   614400  6 vmwgfx,drm_kms_helper,drm_ttm_helper,ttm
fuse                  176128  1
loop                   32768  0
dm_mod                184320  0
efi_pstore             16384  0
configfs               57344  1
ip_tables              36864  0
x_tables               61440  7 xt_conntrack,nft_compat,xt_tcpudp,xt_addrtype,xt_nat,ip_tables,xt_MASQUERADE
autofs4                53248  2
ext4                  983040  1
crc16                  16384  1 ext4
mbcache                16384  1 ext4
jbd2                  167936  1 ext4
crc32c_generic         16384  0
sr_mod                 28672  0
cdrom                  81920  1 sr_mod
ata_generic            16384  0
ata_piix               45056  0
sd_mod                 65536  3
t10_pi                 16384  1 sd_mod
libata                401408  2 ata_piix,ata_generic
crc64_rocksoft         20480  1 t10_pi
crc64                  20480  1 crc64_rocksoft
crc_t10dif             20480  1 t10_pi
crct10dif_generic      16384  0
vmw_pvscsi             32768  2
crct10dif_pclmul       16384  1
crct10dif_common       16384  3 crct10dif_generic,crc_t10dif,crct10dif_pclmul
psmouse               184320  0
crc32_pclmul           16384  0
crc32c_intel           24576  3
vmxnet3                73728  0
scsi_mod              286720  5 vmw_pvscsi,sd_mod,libata,sg,sr_mod
i2c_piix4              28672  0
scsi_common            16384  4 scsi_mod,libata,sg,sr_mod
ben@2480-Z:~$

You can manually start and stop kernel module code with the insmod, rmmod, modprobe commands though this is largely automated in modern Linux systems so it is outside the scope of this course. In fact, with the simple hardware setups provided by virtual machine infrasctructure there is often very little a system administrator needs to do with kernel modules at all (with the possible exception of filesystem kernel modules). However, when running Linux directly on hardware especially with non-standard USB devices or PCI(e) cards you may find it neccecary to build, load, or work with kernel modules in order to take advantage of the hardware.

It is also sometimes useful, for diagnostic purposes, to be able to check what USB or PCI(e) devices the kernel can see and what it knows about them. To check on PCI(e) cards you can run the lspci command as an administrator:

ben@2480-Z:~$ sudo lspci
00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
03:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
ben@2480-Z:~$

Again, on a virtual machine this probably won’t look too exciting. However, it can give us some clues such as that our Ethernet (network) cards are of the VMNXNET3 type which we could then use to search for in the kernel ring buffer (with dmesg which we have used previously) to find more information about such as the device names for configuring them.

If you’re working with USB devices you will probably want to install the usbutils package on the system so you can get similar information about USB devices, go ahead and install that now.

Once you have installed usbutils you should be able to run the lsusb command as an administrator and see some information about attached USB devices:

ben@2480-Z:~$ sudo lsusb
[sudo] password for ben:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 003: ID 0e0f:0002 VMware, Inc. Virtual USB Hub
Bus 002 Device 002: ID 0e0f:0003 VMware, Inc. Virtual Mouse
Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
ben@2480-Z:~$

Again, not very exciting on a typical virtual machine. In fact, many virtual machines don’t even have virtual USB ports at all so you could get no output. However on a physical Linux system with some USB devices attached you can definitely see more:

pi@demosystem:~ $ sudo lsusb
Bus 001 Device 005: ID 1b3f:2008 Generalplus Technology Inc. USB Audio Device
Bus 001 Device 004: ID 10c4:814a Silicon Labs West Mountain Radio RIGblaster P&P
Bus 001 Device 006: ID 0424:7800 Microchip Technology, Inc. (formerly SMSC)
Bus 001 Device 007: ID 0bda:2832 Realtek Semiconductor Corp. RTL2832U DVB-T
Bus 001 Device 008: ID c0f4:05f5 Usb KeyBoard Usb KeyBoard
Bus 001 Device 009: ID 10c4:8108 Silicon Labs USB OPTICAL MOUSE
Bus 001 Device 003: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Microchip Technology, Inc. (formerly SMSC) USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
pi@demosystem:~ $

At some point you may also want to check on the amount of memory a system has and how much of it is in use. This can be done with the free command:

ben@2480-Z:~$ free
               total        used        free      shared  buff/cache   available
Mem:         2014436      621904     1194176        3184      346356     1392532
Swap:         998396           0      998396
ben@2480-Z:~$

This basic output shows the total amount of physcial memory in the system as well as swap space which is a location on the hard drive where system memory information can be stored that hasn’t been accessed in a while. Microsoft Windows sometimes calls this feature virtual memory or a page file. In addition you can see the amounts of used and free memory and the shared, buffer/cache and available memory for starting new applications. However, this information is shown in kibibytes by default which can be a bit less useful in an era of gigabytes of system memory. Try running free -m and free -g to show the same information in mebibytes and gibibytes.

ben@2480-Z:~$ free -m
               total        used        free      shared  buff/cache   available
Mem:            1967         604        1164           3         342        1362
Swap:            974           0         974
ben@2480-Z:~$ free -g
               total        used        free      shared  buff/cache   available
Mem:               1           0           1           0           0           1
Swap:              0           0           0
ben@2480-Z:~$

As you can see, on our system mebibytes may be the most useful measure to use as our systems have 2GB (also known as 1.967 GiB or 1967 MiB) of total physical memory ov which about 604 MiB is in use leaving 1164 MiB (or 1.164 MiB) of available physical memory in my demonstration system.

Linux manages nearly all aspects of the system as if they were files. This means that information such as the amount of free memory, processes running, the kernel itself, and even the CPU capbilities are stored in virtual files somewhere in the filesystem. The location of these virtual files is the /proc "directory".

Change the working directory on your system to /proc and list the "files" in this "directory"

You should see that almost all of the files have a size of 0, have been updated quite recently, and are owned by the root user. What do these virtual files contain though?

Run the head meminfo command to view the first 10 lines of the /proc/meminfo "file":

ben@2480-Z:/proc$ head meminfo
MemTotal:        2014436 kB
MemFree:         1192688 kB
MemAvailable:    1395528 kB
Buffers:           16568 kB
Cached:           307408 kB
SwapCached:            0 kB
Active:            72284 kB
Inactive:         611584 kB
Active(anon):        544 kB
Inactive(anon):   362532 kB
ben@2480-Z:/proc$

Does this look familiar? It should. This virtual file contains information about system memory usage similar to what we got from the free command before. There is actually much more information here (remember we’re only seeing the first 10 lines!) than free shows us and free is just a convenient utility to access information from this file.

In fact, many of the command line utilities we have used to get basic information about the system are just pulling information from the /proc filesystem and reformatting it in a user firendly way. For example, take a look at the /proc/modules virtual file using the cat command. This should also look familiar as the same type of thing you got from lsmod about kernel modules.

The /proc/version "file" should also look familiar as the uname command uses it to get the running kernel version.

In general, you can get more detailed information directly from the kernel by looking at /proc virtual files but they can be a bit harder to decipher than when they are interpreted by other tools. Some new information you should try looking at includes:

There are many more system details in the /proc filesystem including network details and information about all the processes running on the system (where ps and top get there information). In general it’s usually easier to use dedicated command line utilities to get information about the system but it’s good to know that /proc is there in case you need to dig deeper.

It is even possible to enable and disable kernel features on the file by modifying the virtual files in the /proc/sys "directory". However, these changes will only persist until the system is rebooted after which they will get reset. To make changes permanent the sysctl command and the /etc/sysctl.conf file can be used.

As mentioned above Linux tries to treat everything as a file. This even includes devices on the system which are housed in the /dev location on the filesystem. Unlike the /proc location most of the things here do not appear as text files so you can’t read them with tools like cat. Instead, /dev is used more as a path to a device which can be used in other software or configuration files to indicate what device you want to work with.

Change your working directory to /dev and list the "files" in the directory.

Notice there are a lot of devices here but also that a lot of the names are similar like tty, tty0, tty1, tty2, etc. These are all _virtual consoles on the system which are used to allow interaction from a keyboard/monitor or from a service like SSH and because we can have mulitple users working on the system at a time we need multiple virtual consoles.

Here is a table of just a few of the most common /dev devices you may encounter:

Another common system administrator task is to check and see how much disk space is being used on the system. You can check the total amount, used amount and available amound of disk space for all partitions on the system using the df (disk-free) command:

ben@2480-Z:/dev$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
udev              984160       0    984160   0% /dev
tmpfs             201444     776    200668   1% /run
/dev/sda1       15421320 3863460  10752692  27% /
tmpfs            1007216       0   1007216   0% /dev/shm
tmpfs               5120       0      5120   0% /run/lock
tmpfs             201440       0    201440   0% /run/user/1000
ben@2480-Z:/dev$

Here you can see that the main partition on the SATA drive we’re using is /dev/sda1 and it is 27% full. The total, used, and available numbers are given in 1KB blocks though which can be hard to read with today’s much larger partition sizes. Try running df -h which stands for "human readable" format:

ben@2480-Z:/dev$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            962M     0  962M   0% /dev
tmpfs           197M  776K  196M   1% /run
/dev/sda1        15G  3.7G   11G  27% /
tmpfs           984M     0  984M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           197M     0  197M   0% /run/user/1000
ben@2480-Z:/dev$

That’s better we can now see that the /dev/sda1 partition is 15GB in size and has 3.7GB used with 11GB available. Note that there are additional options for df that allow you to change the block size to other sizes too.

What if you want to know what exactly is taking up all that space through? That’s where the du (disk-usage) command comes into play. Don’t run it just yet though as we want to tell it where to start checking file sizes. If you’ve already run it you can stop it by pressing CTRL-C.

Because it takes some time to check the size of all the files on a disk individually you probably want to be specific about where you are checking the sizes. Let’s change our working directory back to your user’s home directory first.

Now run the du command (which should be fairly quick as we don’t have too many files in our home directory). You’ll see a listing of every individual directory and file inside all the directories inside your home directory along with their size. You’ll also see a total usage at the end of the listing.

However, again, the sizes are not that readable so try again with du -h to get human readable sizes.

There’s still a lot of data there to look at so it’s common when trying to identify large files by starting just looking for large directories and then going into those directories and looking for large files. Let’s try an example of this that will show some useful commands to make this easier.

If you start to run out of space and have decided there is nothing you can delete from the system you may decide it’s time to add another hard drive to your system.

Luckily our virtual machines already have a second unused hard drive installed we can practice with to see how this is done. If you run ls -al /dev/sd* you’ll see that in addition to /dev/sda which has a few partitions on it there is a /dev/sdb which is unused so far.

Like almost all operating systems Linux uses partitions to logically divide up a disk into locations which contain a filesystem used to organize data. So, to begin we need to create at least one on the drive which can be formatted with a filesystem.

Before we can store anything on a partition we need to put a filesystem on it by formatting it. While there are many filesystems that can work with Linux all of which have advantages and disadvantages probably the two most common filesystems today are ext4 (which is based on the older popular ext3 and ext2 filesystems) and btrfs so we will focus on those as well as exFAT which is a common cross-operating system filesystem. Other Linux friendly filesystems that merit review in certain use cases are XFS and ZFS as well as things like dm-crypt if you are interested in whole drive encryption.

Current versions of Debian default to using ext4 when installed so let’s start with that on our 5GB partition /dev/sdb1

Another popular Linux filesystem is the btrfs (meaning "b-tree filesystem" and pronounced "butter-fs"). Some Linux distributions have made this the default filesystem as it includes more features (such as actually being able to span a single filesystem across several physical disks or partitions) compared to ext4 but some people view it as a bit less stable or compatible as it is under heavy development and subject to more frequent changes. Let’s try setting up our 2GB partition with a btrfs filesystem.

In some cases you need to read and write files to a device, perhaps a USB thumb drive, which needs to be accessed by other operating systems. Microsoft Windows in particular will not be able to easily read or write files stored on ext4 or btrfs partitions. While Linux can be coerced to work with Windows' NTFS filesystem it is probably also not the best choice. Luckily the exFAT filesystem is almost universally supported by operating systems (though it has very few features) which makes it a reasonable choice for thumb drives and other storage partitions which may need to be read by multiple operating systems. Let’s setup our 1GB partition /dev/sdb3 with an exFAT filesystem.

In order to actually use a partition in Linux we need to mount the partition which is to attach it to some location on the filesystem. This can either be done manually with the mount command or automatically (either during the boot process or based on some event happening such as inserting a CD/DVD or thumb drive).

In this example we’ll be mounting partitions to mount points inside of the /mnt location on the filesystem but this is not a requirement. In fact, you can mount a partition to almost any location in the filesystem. For example, you may want to store all the user home directories on a different partition than evertything else on the system. In that case you could mount a partition to the /home directory so that everything stored in /home is actually being saved to a different partition than all other locations on the system. This makes Linux incredibly flexible with storage, much more so than the one partition per "Drive Letter" system traditionally seen in Microsoft DOS and Windows (note there are advanced commands in Windows which can be used to make it do something more like Linux). You can fairly easily see that a drive or partition is filling up in Linux, add another drive/partition, move files from one directory of the filled up partition to the new partition and mount the new partition to the same location as the old directory. When this is done all the applications still see things in the same logical location on the system (and so configurations don’t need to change) even though they are now physically being stored on a different drive/partition.

Let’s try some simple manual filesystem mounting first to see how this works.

If a partition is going to be regularly used with the system you may want to set it up to automount when the system is booted. This is controlled by the /etc/fstab configuration file.

Close the SSH session

If you are using the Administrative PC in Netlab instead of your own computer as the administrative computer you should also shut down that system in the usual way each time you are done with the Netlab system and then end your Netlab Reservation. You should do these steps each time you finish using the adminsitrative PC in future labs as well.