Linux Networks

20.1. Basic Network Topology

  • Switches forward packets within the internal network
  • Routes forwards packets between networks
  • Data travels through both to reach a destination outside of the LAN
  • Cloud networking is a software defined simulation of a traditional network

20.2. OSI Layers

The Open Systems Interconnection model (OSI model) is a conceptual model that characterizes and standardizes the communication functions of a telecommunication or computing system without regard to its underlying internal structure and technology. Its goal is the interoperability of diverse communication systems with standard protocols. The model partitions a communication system into abstraction layers. The original version of the model defined seven layers.

  1. Physical - bit lvl transmission between network nodes
  2. Data link - handles comms between adjecent nodes [MAC]
  3. Network - best path to reach the destination [IP]
  4. Transport - how data will be sent, validation and security
  5. Session - ability to open, close and manage a session between process and response
  6. Presentation - converts request from application into universal format
  7. Application - interface permitting user to send and receive data through clients and applications

20.3. Network Flow - IP Addresses

Anatomy of an IP address

IP (Internet Protocol) address is the number of a device on a local network. It is a method of uniquely identifying an address (destination) for a specific system. The IP protocol holds 2 different versions:

IPV4: Standard address structure of four “octets” containing numbers between 0-255 for each

  • 4 octets - 32bit
  • each octet: 8bit (0-255)

IPv6: Intended as a replacement for IPv4, consists of a 128 bit hexadecimal number for addressing

  • hexadecimal
  • 8 sections - 128bit
  • each section: 16bit (0-9,a-f)

Leading zeros can be compressed in IP addresses.

Classfull IP addresses are categorized in 5 ranges as specified in the RFC 1918. The values in each determine the total number of hosts that are available in each class (addresses):

Class First Octet Range Private Network Range Subnets
A 0— 0.0.0.0 - 127.255.255.255 10.0.0.0 - 10.255.255.255 255.0.0.0 (few networks - many hosts)
B 10– 128.0.0.0 - 191.255.255.255 127.16.0.0 - 172.31.255.255 255.255.0.0 (som networks - some hosts)
C 110- 192.0.0.0 - 223.255.255.255 192.168.0.0 - 192.168.255.255 255.255.255.0 (many networks - few hosts)
D 1110 224.0.0.0 - 239.255.255.255 (reserved for multicast)    
E 1111 240.0.0.0 - 255.255.255.255    

The Network Mask defines a logical network (called a subnet) that indicates the start and ending of a range of addresses.

Every network has a broadcast address, operating on the address number 255. This is how an IP network sends traffic that can be seen by all hosts on a network.

CIDR: Classless Inter-Domain Routing - use any amount of bits to create networks and hosts.

IPv6 subnetting:

  • Global unicast: Internet scope - routed on the ineternet 2001:
  • Unique local: Internal network scope - not found on internet
  • Link local: Network linked scope - not found on internet FE80:

Network Interfaces

  • You can have multiple IPs on the same network on a single interface.
  • You can have multiple interface cards with the same IP.
  • You can have multiple IPs on multiple interface cards.

When adding multiple interfaces for a single IP address, redundant connections are created. This is called teaming or bonding.

Teaming

Teaming and Bonding are similar, but there are differences:

Teaming Bonding
Support for IPv6 link monitoring Doesn’t require teamd?
Able to work with D-Bus and Unix Domain Sockets Works in a virtual environment
Load balancing for LACP support  
Leverages NetworkManager and associated tools  

Default Behavior:

  • Starting the master (team) interface will not automatically start the port interfaces (slave1).
  • Starting a port interface (slave1) will start the master interface (team).
  • Stopping the master interface (team) also stops the port interfaces (slave, slave2, etc).
  • A master without ports will start static IP connections.
  • A master without ports waits for ports when starting DHCP connections
  • A master with a DHCP connection waiting for ports completes when a port with a carrier is added.
  • A master with a DHCP connection waiting for ports continues waiting when a port without a carrier is added.

Team Runners:

  • broadcast: Transmit data over all ports
  • round-robin: Default, transmit data over all ports in sequence
  • active-backup: One port is used while the other is reserved as a backup
  • loadbalance: Active load balancing and port selection
  • lacp: Implements the 802.3ad Link Aggregation Control Protocol

Link Watchers:

  • ethtool: Default, watches for link state changes (only ethtool should be used whenn running lacp)
  • arp_ping: Monitors availability of MAC addresses using ARP
  • nsna_ping: Neighborhood advertisement and neighborhood solicitation from the IPv6 neighbor discovery are used to monitor neigboring interface

NAT

NAT - Network Address Translation: Network address translation (NAT) is a method of remapping one IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device.

DHCP

Static Dynamic
Dedicated unchanging IP address assigned to the device Dynamic address assigned from a pool of IP addresses within the DHCP scope
Upon setting it, you must also specify the subnet, gateway and DNS host information DHCP will normally provide subnet, gateway and DNS info upon assigning an IP address
Not dependent on DHCP for anything Dependent upon DHCP upon lease expiration
Typically used for important network devices and hosts that require absolute connectivity Typically used for hosts and clients requiring transient connections
  DHCP is used for managing the IP addresses of a large nr of hosts

Consideration for DHCP:

  • Using DHCP creates three potential points of failure in network connectivity: the DHCP host itself, a rogue DHCP host, and DNS
  • The DHCP host becomes an issue if it is misconfigured - especially if you are dependent upon reservations for static-like IP assignment.
  • The potential exists for a rogue DHCP host to present network configurations that could route traffic to a different router for nefarious purposes.
  • If host IPs are changing, you’ll need to rely on FQDNs for host to host communication, placing a dependency on DNS for FQDN to IP resolution.

DHCP makes managing thousands of host Is in a virtual or automated infrastructure on a secure network much easier.

DHCP Leases:

  • The DHCP server will grant a lease on an IP address - assigning it to the host for a period of time
  • The DHCP request and offer are layer 2 broadcasts (similar to ARP requests) and requires a running DHCP client
  • The DHCP server uses UDP port 67, and the Client uses UDP port 68
  • You can use reservations to make sure the same host gets a new lease on the same IP upon expiration

View DHCP address lease information on the host:

    cat /var/lib/dhclient/dhclient. leases

Renew DHCP address lease (3 methods):

  1. dhcpd -k: Kills and restarts the daemon to get another lease
  2. dhclient: After restarting network services, run to get another lease
  3. pump: Legacy utility to obtain a new DHCP address

Lookup DHCP host:

    sudo grep "DHCPOFFER" /var/log/messages

Note: hostnamectl is the command that is used to configure a local system’s hostname persistently.

DHCP Sequence:

DHCP sequence

Configure Network Connections

There are lots of ways to configure network devices. One of the most useful in Red Hat Enterprise Linux is nmcli (The NetworkManager command-line interface utility).

If you the bash-completion package installed, that will help navigate the command.

nmcli will accept shorthand for many of its settings. For exmaple:

    nmcli connection show
    nmcli con show
    nmcli c s

One of the key things you need to know when configuring a network are your interfaces, you can view them in the /sys/class/net/ directory or use the command:

    nmcli dev st

You can view the config for each individual interface in the /etc/sysconfig/network-scripts/ directory.

Create a connection:

    nmcli con add <CONNECTION_NAME> [OPTIONS]

Activate/Disable the connection:

    nmcli con up <CONNECTION_NAME>
    nmcli con down <CONNECTION_NAME>

Modify a connection (connection needs to be inactive to modify):

    nmcli con modify <CONNECTION_NAME> [OPTIONS]

Delete a connection (connection needs to be inactive to delete):

    nmcli con delete <CONNECTION_NAME>

ifconfig is another utility used to view all active interfaces (including loopback adapter). The -a flag allows you to force the system to display ALL interface (active or not) in the report.

  • ifup: Brings up the indicated network interface
  • ifdown: Brings down the indicated network interface

Key fields:

  • ether: Hardware (MAC) address, a 48 bit interface adapter’s physical address
  • inet: The network address assigned to that interface
  • broadcast: Broadcast address for the system’s network
  • netmask: Network mask or logical network segment information

Note: ifconfig is a legacy network command that is being phased out and replaced by the next one.


Another option for configuring your network is the ip command, part of the iproute2 package.

ip is a unified network and routing management command designed to replace the functionality of most of the other commands mentioned above.

Similar to nmcli, you can use bash completion to navigate some of the various settings in this command. ip will also accept shorthand for many of its settings.

For example: provide the same information as the generic ifconfig command above

    ip address show
    ip a s

The main differences between ip and nmcli is that changes made with the ip command are not persistent.

Interface Configuration Files:

/etc/sysconfig/network-scripts is a directory containing a host of scripts responsible for the configuration of all interfaces on the system.

For example: /etc/sysconfig/network-scripts/ifcfg-eth1 is responsible for the configuration (static or DHCP) of the address information for the ETH1 interface on your system.

Changes to the network interface configuration are applied by restarting the network service:

    service network restart

20.4. Network Flow - Routing

The Basics

When a packet is sent between networks, routers inspect the packet’s contents and determine the best path for the packet to take.

The routers determine the best path for the packet using routing tables. The entries on each router’s routing table determine the next “hop” that the packet will go through. For example, a router could decide that the packet belongs on the local network and deliver it there. Or it could decide that the packet needs to go to another router on the way to its final destination.

  • A router is a layer 3 device, functioning at the IP level A router forwards data packets between networks
  • The routing table is a static table mapping of the best path to a network destination
  • The routing table lists destinations and gateways (The destination where network traffic goes that has no other matching route or is not intended for the local system network itself) for the networks the host belongs to
    # example of a routing table

    Kernel IP routing table
    Destination Gateway     Genmask         Flags   Metric      Ref     Use     Iface
    0.0.0.0     172.31.96.1 0.0.0.0         UG      100         0       0       eth0
    172.31.96.0 0.0.0.0     255.255.240.0   U       100         0       0       eth0

There are two primary routing methods: static and dynamic

Static Routing

In a simple network, administrators configure routes statically. Static routing involves manually defining routes on a per-server or per-router basis. Static routes are for traffic that must not (or should not) go through the default gateway.

The default gateway is the destination of ALL traffic whose destination is not on the local system network OR does not have another matching static route configured. It can be set by editing /etc/sysconfig/network or /etc/sysconfig/network-scripts/ifcfg-eth# (where # is the interface number).

  • This practice isn’t scalable. What if you have 100 routers? Or 1000?
  • What happens when one or more routers goes down?
  • Which path is then the best one for network packets to follow?

Example: 172.31.96.0/20 is the server network, and the office network is 10.0.0.0/8. The office network is accessible via a router with the IP of 172.31.96.2 on the server network. A route would need to be added to permit this host to communicate with the office network using that router.

    # example of a routing table

    Kernel IP routing table
    Destination Gateway     Genmask         Flags   Metric      Ref     Use     Iface
    0.0.0.0     172.31.96.1 0.0.0.0         UG      100         0       0       eth0
    10.0.0.0    172.31.96.2 255.0.0.0       UG      100         0       0       eth0
    172.31.96.0 0.0.0.0     255.255.240.0   U       100         0       0       eth0

Dynamic Routing

Dynamic routing solves this problem. In a dynamic routing environment, each router talks to its peers to determine which networks are near which router, then modifies its internal routing table to ensure packets use the smallest number of connections possible to get to their destination. The goal of dynamic routing is to deliver packets quickly via the most direct path.

Dynamic routing comes with a cost. Since the routing protocol operating on the router is responsible for creating, maintaining, and updating the dynamic routing table, dynamic routing has a higher memory and CPU requirement than static routing.

However, the advantages of dynamic routing - such as adjusting routes to avoid network and router failures or to include newly installed routers - far outweigh the higher hardware requirements.

Dynamic routing is typically handled by dedicated devices. Cisco, Juniper, Alcatel-Lucent, and Aruba all build specialized devices for handling dynamic routing.

You can also build your own dynamic router using a package called Quagga.

Quagga is a fork of an older (and unmaintained) package called Zebra, so many of the commands include the old Zebra terminology. Quagga can handle any of the three most popular dynamic routing protocols: Open Shortest Path First (OSPF), Routing Information Protocol (RIP), or Border Gateway Protocol (BGP).

Configure Routing

Command Description
route displays the current routing table and can aad or remove routes as indicated
Parameters Description
add add a gateway/route/destination
default keyword for adding/removing a default gateway
gw short for gateway, all traffic not matching other rules is routed here
[IP address]] the IP of the gateway/route/destination

For example: add a default gateway to the system to 192.168.1.1

    route add default gw 192.168.1.1

For example: route all returned traffic to the loopback adapter (quick and dirty solution for DoS attack from a single IP)

    route add 192.168.10.211 lo

Common Ports/Services

Port number Service
20/21 FTP
22 SSH
23 Telnet
25 SMTP
53 DNS
80 HTTP
110 POP3
123 NTP
139 NETBIOS
143 IMAP
161/162 SNMP
389 LDAP
443 HTTPS
465 SMTPS
514 SYSLOG
636 LDAPS
993 IMAPS
995 POP3S

IP forwarding

IP forwarding is the baility for your host to forward packets to another location and respond, it allows your system to function as a router.

Methods to enable IP forwarding:

  1. echo 1 > /proc/sys/net/ipv4/ip_forward
  2. edit /etc/sysctl.conf and add net.ipv4.ip_forward=1

Note: Method one is not permanent but will take immediate effect, method two requires a reboot (or combined with method one)

20.5. Network Flow - Border Gateway Protocol

Routing protocol used to route traffic across the internet - it’s how the internet works.

  • BGP is a Layer 4 protocol sitting on top of TCP - this makes it simpler than OSPF, as it doesn’t manage what TCP handles
  • There is no discovery; peers that have been manually configured to exchange routing information form a TC connection
  • Organizations can use BGP for true multi-homing their corporate network

  • An ASN (Autonomous System Number) is required to implement BG peering
  • This is a special number assigned by lANA for use primarily with BG that identifies each network on the internet
  • Two routers that have established a connection and exchanged BG information are BG peers, exchanging routing information between them via BGP sessions over TCP

20.6. Network Flow - Address Resolution Protocol (ARP)

ARP is a communication protocol used for discovering the MAC address associated with a given network layer address (IP address). Put simply, ARP is used for mapping an IP address to a MAC address on the network.

ARP Sequence

ARP sequence

  • ARP is necessary to map Layer 3 (IP) addressing to Layer 2 (MAC) addressing
  • This is used for local area network connections
  • Connections outside of the local area network, go through the gateway (ARP request for gateway IP)
  • The ARP table is dynamically updated, and can be viewed using the ip n command.

20.7. Network Flow - DNS

DNS (Domain Name System) is a Layer 7 communication protocol used for discovering the IP address associated with a given domain name. For devices on the network to be reachable, they must be addressable, put simply, DNS is used for mapping a domain name to their respective IP address on the network (internal and external)

DNS Query

  • An FQDN (fully qualified domain name is the complete domain name for a specific host on the network.
  • When a computer wants to initiate a connection to an FQDN, such as www.example.com, it needs to know where the host is on the network.
  • The computer will send a query to the DNS server, asking it to resolve the FQDN to an IP address, and then looks at the routing table to determine where to send the request.
Command Description
dig DNS lookup utility, query DNS records
Option Description
[server] Specify a DNS server to use (optional)
[domain] The specific domain to query for (mandatory)
[record type] Different record types (i.e., NAME, CNAME, MX, etc.) (optional)

DNS Resolution

  1. . - resolved by root DNS server
  2. .com- resolved by TLD servers
  3. domain.com - resolved by name servers
  4. www.domain.com - resolved by name server or other auth server within domain
  5. ip address received and connection can be made

/etc/nsswitch.conf determines the order in which local name resolution vs. remote name resolution queries take place.

For example:

    hosts:      files dns myhostname

Would cause the system to check /etc/hosts for name resolutionmatches BEFORE checking nameservers defined in /etc/resolv.conf.

Note: files in /etc/nsswitch.conf refers to /etc/hosts, which is the local DNS.

Configure DNS

  • /etc/resolv.conf - This config file is used to determine which hosts to use for DNS queries and shows you which DNS servers you are using.
  • /etc/hosts - Used for statically mapping IP addresses to hostnames
  • /etc/nsswitch.conf - Controls the order that resources are checked for resolution (see DNS Resolution).

You can modify a DNS entry with the following command:

    nmcli con mod <CONNECTION_NAME> ipv4.dns <IP_ADDRESS_DNS_SERVER>

If you want to add another DNS server, by default, it overwrites the current server.

  • To add a second DNS server, you can use +ipv4.dns.
  • To remove a second DNS server, you can use -ipv4.dns.

Note: this command is only written upon a network restart, so there will be no changes in /etc/resolv.conf yet. You are able to view them without restarting in the /etc/sysconfig/network-scripts/ifcfg-<INTERFACE> script.

20.8. Network Flow - TCP/UDP and the Basic Handshake

TCP is the method by which all transactions between IP addresses (of any version) are communicated. Defines a system of transmissions and acknowledgement to verify traffic arrives and can be assembled in the correct order.

  • Transmission Control Protocol
  • TCP exists in Layer 4, along with UDP
  • Highly reliable connection protocol through Positive Acknowledgement with Retransmission (PAR)
  • Data is re-sent if an acknowledgement isn’t received
  • The layer 4 data is referred to as segments, each segment contains a checksum for verification upon receipt
  • If verification succeeds, an acknowledgement is sent. If verification fails, the receiver discards that segment and waits for retransmission

TCP handshake

UDP is ften considered “complementary” to IP, but is a “stateless” connection. No error checking or retransmission of packets takes place, even if the transmission of the packet failed.

20.9. Network Flow - ICMP

The Internet Control Message Protocol is designed for networking devices (routers, intelligent switches, firewalls, etc.) to send error messages. In addition, it can perform queries around network service availability (as in the case where the ping command is used to test whether and address responds to a request).

20.10. Network Flow - VPN

A VPN connection is an extension of a private network, using the IP connectivity of the internet to connect remote clients to remote sites in an encrypted private connection.

  • Secure private network traffic over an insecure public network
  • Permit connections to an internal corporate resource from a remote location
  • Connect two separated private networks together
  • The routing table lists destinations and gateways for the networks a host belongs to

Site to Site VPN

A site-to-site VPN connects two parts of a private network (or two private networks). This allows an organization to have routed connections between separate offices, or with other organizations, over the Internet. A routed VPN connection across the Internet logically operates as a dedicated Wide Area Network (WAN) link.

Remote Access VPN

A remote access VPN connection is made by a remote access client. A remote access client is a single computer connecting to a private network from a remote location. The VPN server provides access to the network resources where the VPN server is connected. The packets sent across the VPN connection originate at the VPN client.

VPN Tunnel

Tunneling permits the encapsulation of one type of protocol packing within the datagram of a different protocol, for instance, sending TCP/IP traffic over the internet.

For PPT and Layer Two Tunneling Protocol (L2TP), a tunnel is similar to a session. Each end of the tunnel must agree to the tunnel connection, and will negotiate configuration variables such as address assignment, encryption, and compression parameters. The mechanism used to create, maintain and end the tunnel is the tunnel management protocol.

Only after the tunnel is established can data be sent. When the tunnel client sends network data to the tunnel server, the tunnel client appends a tunnel data transfer protocol header to the payload. The client then sends the encapsulated (and usually encrypted) data to the tunnel server. The tunnel server accepts the data, removes the tunnel data, and forwards the payload to the destination network within the VPN.



VI. Network - Monitoring and Troubleshooting

To be effective in troubleshooting the system and gather/intepreting diagnostics, you have to adopt the troubleshooting mindset.

  1. Remain calm.
  2. Gather information:
    • Define the issue and reproduce.
    • Expand on the issues by gathering logs and other data.
  3. Hypothesis:
    • Guess the problem.
    • Test the problem.
  4. Fix.

Gathering System Information

Command Description
uptime Displays load information, from left to right: time, uptime, users, and the load average in 1-, 5-, and 15-minute increments

Command Description
df Reports diskspace - See Storage Management - Disk Space Usage.

Command Description
free Displays free and used memory

Command Description
lsof Show open files - See Who is logged in?

Command Description
ss Dump Socket statistics
Option Description
-l Show listing
-p Show processes
-t Show TCP

Command Description
ps Show processes for user - See Running Operating Systems - Processes.

21.1. Monitoring the network

The iptraf-ng command installed with a package by the same name provides a lot of useful information.

It can also log the data it collects allowing you to go through the logs and investigate issues at a later time. This does take a significant amount of disk space, however.

Another option is in the EPEL repo - vnstat.

vnstat is both a service and an interactive program. While the statistics collected aren’t as in depth as iptraf-ng, it can help analyze spikes in activity and identify spikes at times there wouldn’t normally be one.

vnstat can also output information in JSON format to be consumed by another reporting tool.

There are 5 layers of communication that a packet goes through to get to a destination:

  1. Physical layer - cabling (ethernet, fiber, or the air for wireless communication)
  2. Data Link layer - switches, access points
  3. Network layer - routers, Linux kernel
  4. Transport layer - TCP, UDP
  5. Application - HTTP, SMTP, SSH, FTP, etc.

If you’re having a routing problem, trying to troubleshoot the application isn’t going to help, you will need to work within the context of the network and communication layers.

The physical layer

Probably the easiest layer to troubleshoot - is it plugged in?

If you can ping the interface, it’s not a physical layer issue. Another way to check is to run ip a s | grep UP and see if the interface is connected or not. If you see something similar to the following for the relevant interface, it’s not a physical layer issue.

2: enpOs3: <BROADCAST,MULTICAST,UP,LOWER_ UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

Similar to the physical layer, this one is relatively easy to troubleshoot for a system administrator. Can you ping the interface? If something on this layer is having a problem, it’s likely that the problem isn’t going to be limited to a single server.

Switches often use some of the same hardware for multiple ports, so a hardware failure will affect multiple servers. Similarly, a configuration error on the switch will typically mean multiple connections aren’t working.

The Network and Transport layer

This is the layer where you need to start looking at the configuration on your server.

  1. Check the routing table (ip route).
  2. Check the firewall (iptables -L or firewall-cmd -info-zone=public) - This includes making sure UDP or TCP (whichever is required is oper
  3. Use the nmap utility to see what ports are open (or Listening) on the problem server
  4. You can use tcpdump for protocol specific troubleshooting to see if the packets are even getting to the server (it’s possible there’s a firewall somewhere else that needs to be configured)
Command Description
ping utility to test the response from a particular network address
Option Description
-n [#] ping indicated number of times

Note: ping6 is the ipv6 equivalent to ping.


Command Description
netstat print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships
Option Description
-a Show all sockets on active interfaces
-c Refresh stats every 1 second
-p Shows name and PID for each socket
-t Show TCP stats
-r Show the routing table
-n Do not attempt name resolution (IP only)

Command Description
traceroute utility to determine the distance (in hops) between your system and a desired endpoint, as well as the response time of each hop along the way
Option Description
-n do not attempt name resolution for each hop, IP only

Note: Only privileged users can make use of traceroute.

Note: traceroute6 is the ipv6 equivalent to traceroute.


Command Description
tracepath utility to determine the distance (in hops) between your system and a desired endpoint, as well as the response time of each hop along the way
Option Description
-n do not attempt name resolution for each hop, IP only

Note: Amm users can make use of tracepath.

Note: tracepath6 is the ipv6 equivalent to tracepath.

The Application layer

This is where we start to look at things like potential DNS issues (dig, nslookup, host) or application misconfigs.

/var/log/messages may have relevant information for you.

    # application specific logs only
    journalctl -u servicename.service

Different applications have different troubleshooting steps.

For example, httpd might be configured to look for index.hmtl instead of index.html which is causing it to not respond correctly.



VI. Network - Security

22.1. Determine Current Security State of a System

Who is logged in?

Command Description
who Lists out the currently logged in users on a system

Command Description
who This command will also list the currently logged in users, but can also show what processes they are currently running

Command Description
last This command shows a listing of user who were logged into the system (but are now logged out)
Option Description
-f /Var/log/btmp List only the users with failed login attempts

Command Description
lsof This command can be used to determine what file are currently opened on the system
Option Description
-u <USERNAME> List only files created by a particular user

Note: Network ports are also shwon in the output because they are considered by the system as files. This means that open network ports can be viewed with this command as well.

22.2. Network Firewalls

Firewalls

Netfilter

The piece of software in the kernel that handles firewall interactions is called netfilter. It is a set of hooks that allows kernel modules to register callback functions with the network stack. A registered callback function is then called back for every packet that traverses the respective hook within the network stack. There are a few different ways to interact with netfilter from user space, but we’re primarily going to cover firewall-cmd and firewalld.

netfilter is primarily used for:

  • Packet filtering
  • Network address translation
  • Port translation

Firewall Persistence

There are 2 configuration areas inside firewalld.

  1. Runtime
  2. Permanent

Making a change on one does not require making a change in the other.

  • Making a runtime change takes effect immediately, but doesn’t persist through reboots or firewalld restart.
  • Making a permanent change only takes effect on reboot or firewalld restart.

firewalld comes with multiple zones already built in. A zone is just a convenient grouping of rules and things to apply those rules to.

The “home” zone for example, could have a source of 192.168.0.0/24 and allow things like samba, nfs, ntp, and so on.

    firewall-cmd -get-zones
    firewall-cmd -get-default-zone

Once you know what zone you’re working with, you can look at the rules that are being applied.

    firewall-cmd --zone=home --list-all
    firewall-cmd --zone=home -add-service=http
    firewall-cmd --zone=home -add-port=80/tcp
    firewall-cmd --add-source=192.168.0.0/24

Note: These commands are made in the runtime config area of firewalld. To make the changes permanent, add --permanent flag to your commands.

iptables

Tables:

  • Filter Table: Used for determining if a packet is permitted to continue or be denied
  • NAT Table: Used for performing network address translation rules by determining how to modify a packet’s source or destination address to effect routing
  • Mangle Table: Used for altering the IP headers of a packet in order to modify TTL, hops, etc
  • Raw Table: Used for opting out of connection tracking
  • Security Table: Used for setting SELinux security context values on packets or connections

Rules:

  • Matching: Determine what disposition a packet must have in order to be matched against a target.
  • Targets (action): Normally divided into terminating and non-terminating (the chain of evaluation). Jump targets are non-terminating and move the evaluation to a different chain.
    • ACCEPT
    • REJECT
    • DROP
  • Examples:
    iptables -t filter -A INPUT -p tcp --dport 22 -j ACCEPT
    iptables -I INPUT 5 -s 10.0.1.0/24 -j REJECT

States:

  • NEW: A new packet not associated with any existing connection
  • ESTABLISHED: Established traffic (SYN/ACK)
  • RELATED: Packets associated with a connection already in the system, but not an existing connection
  • INVALID: Unrouteable or unidentifiable packets not associated with an existing connection or suitable for a new connection
  • UNTRACKED: Packets set in the raw table chain to bypass connection tracking
  • SNAT: Source modified by NAT
  • DNAT: Destination modified by NAT

22.3. SELinux

Access control is an important part of Linux system administration. There are two main approaches to file permissions in Linux: Discretionary Access Control (DAC) and Mandatory Access Control (MAC).

DAC is what a normal, unconfined process uses. With DAC, a user has full permissions over its objects. For example, the owner of a log file can modify it or change the read/write permissions for the file.

MAC enables significantly more fine-grained controls. It can restrict the owner of a log file to only append operations, for example. It can also be used to limit the capabilities of a regular process (for example, by denying debugging or networking capabilities).

SELinux Security Enhanced Linux, is a set of rules that define what process can have access to what files on a system. It does that by labeling every file port and socket with a context.

SELinux Policies

There are two main SELinux policies:

  • targeted (the default policy)
    • Only targeted processes are protected by SELinux.
    • Everything else is unconfined.
  • mls (Multi-Level Security)
    • Typically only used when the absolute highest security is needed.
    • Can be extremely complex to set up.

SELinux States

SeLinux has 3 different states:

  • Enforcing: SELinux security mode is enforced
  • Permissive: SELinux prints warnings but allows access that would normally be denied.
  • Disabled: no SELinux policies are loaded. This mode is not recommended in a production environment, and requires reboot after changing the setting.

Switching between enforcing and permissive can be done immediately on the command line with:

    getenforce
    setenforce 0|1

How does SELinux Work

Two most important SELinux concepts:

  • Labeling
    • Files, processes, ports, etc. are all labeled with an SELinux context.
    • Where possible (files, directories), these labels are stored as extended attributes on the filesystem.
    • For everything else, the kernel manages these labels.
  • Type Enforcement
    • Type enforcement is the part of the policy that says, for instance, “A process running with the label httpd_t can have read access to a file labeled httpd_config_t.

Booleans

A Boolean is a conditional rule that allows runtime modification of the security policy without having to load a new policy.

Booleans handle everything from basic permissions (like whether or not the FTP server can access home directions) to more advanced onces (like whether or not httpd can use mod_auth_ntlm_winbind).

Get booleans for a particular policy:

    getsebool <POLICY_NAME>

Set booleans for a particular policy:

    setsebool <POLICY_NAME> (on|off)

Note: to make setsebool persistent, add the -P flag at the end of the command.

Managing SELinux Contexts

View SELinux contexts with ls -Z.

There are literally hundreds of built in SELinux context labels:

    semanage fcontext -l    

To see what context a given process is running as:

    ps auxZ | grep <PROCESS_NAME>

Add a new context:

    semanage fcontext -a -t <TYPE> '/<DIRECTORY_NAME>(/.*)?'
    restorecon -r /<DIRECTORY_NAME>

SELinux Causes Errors… Why not just disable it?

When you’re driving our car and you hear noises coming from under the hood, do you turn up the volume on the radio and pretend nothing is happening?

Disabling SELinux because there are errors is like ignoring a problem with your car. The problem isn’t going to go away on its own, and it will probably get worse over time. Luckily, fixing issues with SELinux is much easier than fixing a broken-down car.

SELinux has a “permissive” mode where access isn’t denied, but errors are logged so you can fix the problem without interrupting services.

Typically, SELinux errors are caused by label issues. Less commonly, there’s a problem with the policy - maybe a boolean needs to be tweaked, orwe need to write a custom module.

If the problem stems from a package shipped by your software vendor, submit a bug report! These kinds of issues won’t get fixed until the vendor knows about them.

On rare occasions, vou’ll encounter errors because someone is actively trying to gain access to your system. In that case, MAN THE BATTLE STATIONS!

Troubleshooting

One of the best ways to troubleshoot SELinux issues is to instal setroubleshoot-server.

Once that’s installed you can run the following command to get troubleshooting information and suggestions:

    sealert -a /var/log/audit/audit.log

22.4. AppArmor

AppArmor has two modes:

  • Complain/Learning: Violations of the MAC are logged but not prevented.
  • Enforced/Confined: Violations of the MAC are prevented and logged.

AppArmor profiles are set per binary. That means that every protected binary has an entry in /etc/apparmor.d/. Profiles are named using the path to the binary, with . replacing /. For example, the profile for /bin/ping would be /etc/apparmor.d/bin.ping.

aa-complain and aa-enforce are used to set specific profiles (ping, httpd, etc.) to their respective modes.

22.5. Service Management Security

Determining how/when a service is running (always on, running on demand, or disabled).

Note: Neither of these are generally installed by default on modern distributions.

inetd

inetd is a legacy daemon/service for providing system services “on demand” or as needed. The configuration for inetd can be changed by editing /etc/inetd.conf, it contains a single line for each control.

For example: Sample configuration for telnet, the following order of fields

    telnet stream tcp nowait root /usr/sbin/in.telnetd in.telnetd
Field Description Example
Service name The name of the service to control telnet
Socket type Values can be stream, dgram, rdm, seqpacket stream
Protocol /etc/protocols file defines the values, generally either TCP or UDP tcp
Wait/nowait Wait for single threaded services, nowait formulti-threaded services nowait
User/group The group or user account the service will run as root
Path to command Full path to the application/service to run /usr/sbin/in.telnetd
Arguments Anything that may be required to complete the service configuration in.telnetd

xinetd

xinetd is a replacement for inetd allowing more granular control of services. The configuration for xinetd can be changed by editing /etc/xinetd.conf (or /etc/xinetd.d), each file controls a service.

The request will come to the daemon, which will check for the service type and port and then scan for the appropriate service configuration file in /etc/xinit.d.

Field Description
cps = [#] [#] First number limits the connections per second to the service, second number is the delay before more connections are answered
instances = [#] The total number of daemons allowed at any time

Typical Services

  • finger
  • imap
  • rsh/rsync
  • telnet

For example: rsync config file from /etc/xinet.d

    service telnet
    {
        flags = REUSE
        socket_type = stream
        wait = no
        user = root
        server = /usr/sbin/in.telnetd
        log_on_failure += USERID
        disable = no
    }
Field Description
service Name of the service
disable yes (disable and do not accept connections) and no (do not disable and allow connections)
flags Varies by service
socket type Values can be stream, dgram, rdm, seqpacket
wait/nowait wait for single threaded services, nowait for multi-threaded services
user/group The group or user account the service will run as
server_args Arguments to send to the service
log_on_failure Log failures including the USERID

Note: Run a service from xinetd as another user other than root.

TCP Wrappers

  • inetd: using only /etc/hosts.allow and /etc/hosts.deny as parameters of tcpd
  • xinetd: the library libwrap.a allows those services to use /etc/hosts.allow and /etc/hosts.deny

Note: Order of precedence for hosts.allow and hosts.deny

  • hosts.allow is read first, matches are allow and hosts.deny skipped (entirely)
  • Changes to either hosts.allow/deny take immediate effect on save
  • Sequential read, multiple entries for the same service will cause only the first match to apply
  • If files do not exist, no rules apply
  • Different options can be added rather than just denying service in.telnetd: 10.1.10.0/24 : twist /bin/echo "Service 404 - Service Not Found"

For example: allow (or deny) users of the 10.1.10.0/24 network access to the telnet service (depending on the file it exists in)

    [service/daemon]: [host(s)]: [option]: [option]
    in.telnetd: 10.1.10.0/24

Methods to use in defining policies:

  • Deny by default: Denies ALL hosts access to ALL services (ALL:ALL), used as a fallback in hosts.deny since matches in hosts.allow will override
  • Allow by default: Implicitly trust everyone and provide access to everything (Security risk for obvious reasons)
  • Mix: Allow and deny selectively

22.6. Securing Data with Encryption

gnupg

  • GnuPG Privacy Guard
  • Public and private key creation for encrypting data

gpg

Command Description
gpg Utility for working with keys
Option Description
--gen-key Will prompt you for key type (RSA would be the general default), follow prompts to provide the data in creation of your keypairs
--import [key] Import the indicated key (public key from user who will be sending you a file)
--list-keys List all keys you have imported

Entropy: the “seed” that is used to generate encryption keys, it has to do with the randomness of certain events on your system (generating it correctly can take a long time on large keys).

Command Description
rngd Utility for generating entropy
Option Description
-a Generate a text public key to provide
-o [filename] The output file

Note: When generating keys, be sure you have logged into the system directly with the username and password or the environment or GPG will not be setup without manual configuration. Once complete, the key name will be listed on the output, NOTE that value.