Fail2BAN with Dell Enterprise SONiC

 

SONiC with Fail2Ban Container

Fail2Ban offer a security enforcement to protect against malicious attack based on port/services identification and brute force authentication.

Disclaimer

The Fail2ban is a free software, developed and maintained by the open-source community. It falls under the terms of the GNU General Public License as published by the Free Software Foundation. DELL do not provide any support related to the Fail2ban software.

The Goal

To reduce the risk of brute force attack over SSH (Secure Shell) or a REST API, Fail2Ban can be used to protect against multiples fail authentication based on bad login/password. Fail2ban uses iptables and a specific table to manage a dynamic filtering.

About Fail2Ban and SONiC

Enterprise SONiC provides an audit file (located in /var/log/audit.log) that logs all the changes and all the events with an associated flag (Warning, Failed, Success)

Fail2ban parses log files and bans IPs that show signs of the malicious behavior — too many password failures, seeking for exploits, etc. Generally, Fail2Ban is used to update iptables rules to reject the malicious IP addresses for a specified amount of time, although any other arbitrary action (like sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (Apache, courier, ssh, etc).

Fail2Ban can reduce the rate of incorrect authentications attempts, however it cannot eliminate the risk associated with weak authentication.

About this whitepaper

This document describes an example of a typical and basic configuration of the fail2ban container from the Docker Hub but a lot of variation in term of rule and usage are possible. We will see how to add local rules to parse the audit log to identifying SSH login failures and ban the IP source from where the failures come. Fail2ban allows to define some variables as max failures, time to ban, delay before unban, identify recursive request and some other capabilities. If you need further configuration information, the fail2ban wiki is accessible on GitHub at https://github.com/fail2ban/fail2ban/wiki.

The process installation was successfully test with:

Dell Enterprise SONiC

3.5.0

4.0.5

4.1

4.4.x

Iptables, Nftables and Enterprise SONiC

Entreprise SONiC 4.0.5 and later is compiled with nf_table support whereas SONiC 3.5 doesn’t not have this option enabled so some differences exist between those two versions in term of filtering behavior.

Nftables is a recent implementation of Linux Kernel packet classification providing more flexibility and native NAT (Network Address Translation) support as well as a Netlink API for third-party applications. For further detail on nftables check the Wikipage of the project (What is nftables? – nftables wiki)

By default, Fail2Ban uses the legacy mode of iptables but does support nftables so depending on the SONiC Release Fail2ban rules visibility would be different :

  • With SONiC 3.5, Fail2Ban can keep using legacy mode. Rules generated by FAil2Ban would be visible from the SONiC Linux shell by using the iptables command
  • With SONiC 4.0.5 and later,
    • if Fail2Ban is left with default settings (legacy mode), rules generated won’t be visible by the iptables command but by the iptables-legacy command. This would add extra complexity as all rules cannot be seen in one shell command.
    • If Fail2Ban is configured with nf-table support, all rules (including the ones generated by Fail2Ban) would be visible by using nft command from the SONiC Linux shell

So as a best practice, with SONiC 4.0.5 and later, nftable support should be enabled within the Fail2Ban container (please refer to section 3.1 for detail process)

Note:By default, SONiC CLI is unidirectional in term of synchronization with iptables/nftable. For instance, if you create a ACL from the SONiC CLI, it would be visible from the Linux shell iptable nevertheless iptables/nftable rules created from the Linux shell won’t be visible from the SONiC CLI. So the rule generated by Fail2Ban won’t be visible from the SONiC CLI. Therefore, best practice would be to use Linux shell iptables/nftable to check and diagnose rules insertion. .

Scenario and goal

To reduce the risk of brute force attack over SSH or a REST API, Fail2Ban can be used to protect against multiples fail authentication based on bad login/password. Fail2ban use different specific sub-tables (from the iptables/nftable tool) to manage dynamic filtering. The container manipulates the same kernel packet’s filtering tables than the SONiC host.

Deploying the Container

We assume that SONiC 4.1 is already configured with an internet access over the management interface (IP + default route) and a public nameserver (DNS)

From the sonic-cli, we use the SONiC image installation manager, called “TPCM”, it allows to install and manage third-party container images into the Enterprise SONiC. Please note that TPCM is available starting with SONiC 4.1 on onward. We will cover Fail2Ban installation without TPCM (for SONiC release below 4.1) in the annexe 1.

In this document, we use the official Fail2ban container available on docker hub portal (https://hub.docker.com/r/linuxserver/fail2ban)

Connection to the Enterprise SONiC switch

After login at the prompt type sonic-cli to launch the CLI (Command Line)

Installing the Fail2ban container by using TPCM commands

Use the tpcm command to launch installation of the container. To get more information see the cli guide Enterprise SONiC Distribution by Dell Technologies Management Framework CLI Reference Guide Release 4.1.0 | Dell US

TPCM args options are the same as the docker one: they must be between quotation marks

tpcm install name fail2ban pull linuxserver/fail2ban args “-v /var/log:/var/log:ro \
–net=host \
–cap-add=NET_ADMIN \
–cap-add=NET_RAW \
-e PUID=1000”

Type “exit” after the container installation

Let’s check the container and the host iptables status. Note that no ACL (Access Control List) has been configured on the SONiC host

Docker Fail2ban Status

From the linux shell, run the following command :

sudo tpcm list

SONiC packet filter rules

sudo iptables -S

Result for the Enterprise SONiC 4.1

Because Enterprise SONiC 4.1 use Nftables, we can show nftables rules with the command

sudo nft list ruleset

Configure Fail2ban

let’s connect to the container in interactive mode to add additional filter and jail rules.

sudo tpcm exec name fail2ban

Packet filtering

Let’s first compare the kernel packet filtering tables content from the container and from the host

Result with Enterprise SONiC 4.1, of iptables -S command from the container:

Iptables -S

Result with Enterprise SONiC 4.1, of iptables -S command from the host:

The container does not “see” the kernel’s packet filtering table.

Let’s clarify this behavior:

Since the Enterprise SONiC 4.0.5, iptables is compiled with the nftables support but, by default, Fail2ban container is built without nftables support. Thus, to view all the kernel packet filtering rules, nft command (from the container) must be used instead of the iptables one.

Result with Enterprise SONiC 4.1, of nft list ruleset command from the container:

nft list ruleset

Result with Enterprise SONiC 4.1, of nft list ruleset command from the host:

The rules displayed with the nft tool (using the command nft list ruleset) are the same from both, the container and the host

Please, note the kernel’s packet filter is shared between the host and the container.

Add support for nft in the Fail2Ban container (for Enterprise SONiC 4.0.5 and above)

To make Fail2Ban uses nftable, we need to add the command “banaction = nftables-multiport” in jail.local config file located in the /etc/fail2ban directory.

vi /etc/fail2ban/jail.local

[DEFAULT]

banaction = nftables-multiport

chain = input

This change allows Fail2ban to add subset of rules in the kernel packet filter table by using the nftables tools, and allow the visibility of the rules from the host by using the nft command

Add Fail2ban filters rule

In the Fail2ban container, the filter rule are located in the /etc/fail2ban/filter.d/ directory and must be named as <myfiltername>.local. In this example, we use the name audit-auth.local

In this white paper, we define a “failregex” allowing to identify source IP with multiples failed SSH attends.

vi /etc/fail2ban/filter.d/audit-auth.local

[INCLUDES]

before = common.conf

[Definition]

failregex = ^.*Failed password for (.*)from <HOST>

ignoreregex =

This regex defines the match expression that should be looked for in the /var/log/audit.log file

Add Fail2ban jails rule

After creating the filter, we define the jail ²in the /etc/fail2ban/jail.d/ directory. The jail must be named <myjailname>.local. In this example we use the name audit-auth.local

vi /etc/fail2ban/jail.d/audit-auth.local

[audit-auth]

enabled = true

logpath = /var/log/audit.log

maxretry = 3

The jail config file allows to parse the /var/log/audit.log and block the IP after 3 connection failures

Reload the Fail2ban:

From the container, we reload the jail and the filter. After the reload, fail2ban creates filter tables dynamically

fail2ban-client reload

From the /config/log/fail2ban.log, we see the creation of the new jail

It is also possible to restart the fail2ban server by using the command

fail2ban-server restart

Test and validation

From the container, we display the logs located in /config/log/fail2ban/fail2ban.log and in /var/log/audit.log

tail -f /var/log/audit/log /config/log/fail2ban/fail2ban.log

From a remote host, we initiate several connections with a wrong login/password.

Connection test with wrong login or password

The IP of the host, with failed authentication is identified and blacklisted. A rule is added dynamically by the container. Fail2Ban create dynamically a table named “f2b-audit-auth”, with the list of the IP banned. Fail2Ban also adds a rule in the default INPUT table to block all connection of any IP from the “f2b-audit-auth” table.

Log from Fail2ban and audit.log

Fail2ban matches the 3 failed connections with the wrong password and retrieve the source IP to ban those IP.

Show the added rules from the container

Iptables -S

Or, by using the nftables tools

nft list table inet f2b-table

If we try to connect again from the server, the connection is now rejected.

Show packet filter table from the host

If we check the iptables from the SONiC Enterprise, the result is different between versions

Iptables result with the Enterprise SONiC 4.1 (from the host)

When the Fail2Ban rules are added since the Enterprise SONiC 4.1 we use the iptables-legacy command to see all tables and rules.

Or by using the nftables command

Conclusion

Fail2Ban can identify login, password, certificate failure attempts. The user can adapt the fail regex rules strategy to capture and support others authentication issues.

Using Fail2ban as container in SONiC is a powerful and flexible tool that would improve the security of any inbound connection to the SONiC device like ssh, https, snmp, gRPC or any other TCP/UDP based connection assuming a log file exists.

Annexes

Dell Enterprise SONiC 3.5.0 and 4.0.5 install notes

Docker status

To show the docker status with Enterprise SONiC 3.0.5 or Enterprise SONiC 4.0.5

docker ps -f name=fail2ban

Connecting to the Fail2Ban container in interactive mode

Connection to the container with Enterprise SONiC 3.0.5 or Enterprise SONiC 4.0.5

sudo docker exec -it fail2ban /bin/bash

Iptables status with Enterprise SONiC 3.5.0

Iptables status before Fail2Ban installation

Iptables status in the container before IP Ban

Iptables status in the container after Fail2Ban ban an IP

Iptables status from the host.

With the Enterprise SONiC 3.5, iptables result is the same from the host or the container

Upgrade fail2ban

Use the tpcm command to launch installation of the container. To get more information see the cli guide Enterprise SONiC Distribution by Dell Technologies Management Framework CLI Reference Guide Release 4.1.0 | Dell US

sudo tpcm upgrade name fail2ban pull linuxserver/fail2ban -y

Install Fail2ban by using RestAPI

Here the request body to deploy the TPCM via REST API Command

Install Fail2ban by using remote_tpcm

For a prof of concept i develop a small python tool allowing to deploy the container remotely by using rest-api. This tools is available on SONiC_TPCM (github.com)

OS10 Basic Config

Exemple de configuration simple

Soit un domaine VLT interconnecté en L2, via une interco LACP à un autre domaine VLT.

Pas de routage activé sur les équipements. On considère que le SRV1 et SRV2 ont des vlans en communs.
On aura donc un trunk avec les n Vlans nécessaires déclarés sur le PO1 (attachement du serveur) mais aussi sur le PO100 qui sert à l’interco des deux domaines VLT.

On commence par interconnecter les deux ports OOB pour avoir un lien de HeartBeat entre les deux swicths du même domaine VLT

SW1SW2
interface mgmt1/1/1
 no shutdown
 no ip address dhcp
 ip address 10.10.10.1/30
 no ipv6 enable
!		interface mgmt1/1/1
 no shutdown
 no ip address dhcp
 ip address 10.10.10.2/30
 no ipv6 enable
!
Ceci n’est pas obligatoire, mais est une bonne prarique.

Déclaration du VLT entre SW1 et SW2

SW1SW2
vlt-domain 1
 backup destination 10.10.10.2
 discovery-interface ethernet1/1/13
 primary-priority 4096
 vlt-mac xx:xx:xx:xx:xx:xx		vlt-domain 1
 backup destination 10.10.10.1
 discovery-interface ethernet1/1/13
 primary-priority 8192
 vlt-mac xx:xx:xx:xx:xx:xx
remplacer la vlt-mac, pas une addresse mac forgé, cette ligne n’est pas obligatoire, mais elle fait partie des bonnes pratiques.
Si vous n’utilisez pas le mgmt port, ne déclarez pas de backup destination.

On rajoute les vlans nécessaires

SW1SW2
interface vlan10
 description “VLAN10”
 no shutdown		interface vlan10
 description “VLAN10”
 no shutdown
interface vlan12
 description “VLAN12”
 no shutdown		interface vlan12
 description “VLAN12”
 no shutdown

On configure les interfaces d’attachement au serveur SRV1 (mais ça pourrait être un switch). Dans le cas d’un serveur attention au type de teaming/bonding, dans le cas présent on est sur un teaming de type switch dépendant.

On en profite pour creer le Port-Channel 1, et on laisse passer les VLANs 10, 12 dans trunk.

SW1SW2
interface ethernet1/1/1
 description “Vers SRV1_Port1”
 no shutdown
 channel-group 1 mode active
 no switchport
 flowcontrol receive off		interface ethernet1/1/1
 description “Vers SRV1_Port2”
 no shutdown
 channel-group 1 mode active
 no switchport
 flowcontrol receive off
interface port-channel1
 description “PO1_SRV1_P1”
 no shutdown
 switchport mode trunk
 switchport trunk allowed vlan 10,12
 vlt-port-channel 1		interface port-channel1
 description “PO1_SRV1_P2”
 no shutdown
 switchport mode trunk
 switchport trunk allowed vlan 10,12
 vlt-port-channel 1

On configure ensuite les interfaces et le port channel à destination du second domaines VLT. On laisse passer dans le trunk du port channel 100 les Vlans 10, 12.

SW1SW2
interface ethernet1/1/10
 description “Vers VLT2_SW1_Port10”
 no shutdown
 channel-group 100 mode active
 no switchport
 flowcontrol receive off		interface ethernet1/1/10
 description “Vers VLT2_SW2_Port10”
 no shutdown
 channel-group 100 mode active
 no switchport
 flowcontrol receive off
interface port-channel100
 description “PO100_VLT2_P10”
 no shutdown
 switchport mode trunk
 switchport trunk allowed vlan 10,12
 vlt-port-channel 100		interface port-channel100
 description “PO100_VLT2_P10”
 no shutdown
 switchport mode trunk
 switchport trunk allowed vlan 10,12
 vlt-port-channel 100

On répéte l’action en adaptant sur le second domaines VLT.

Attention les domaines VLTs doivent avoir des ID unique, donc la déclaration du second domaine VLT se fera comme suit, sur les deux switchs du VLT domaine 2

vlt-domain 2
backup destination xx.xx.xx.xx
discovery-interface ethernet1/1/13
primary-priority 4096 [8192]
vlt-mac xx:xx:xx:xx:xx:xx

Management ACL on N-Series

Management ACLs filter packet use to manage the switch.

First we do create a specific access-list with the keyword management

console(config)# management access-list AdminList

You can restrict by source vlan, Ips, subnet for a specific services

The list of service is restricted to:
– Telnet
– SSH
– Http(s)
– TFTP
– snmp
– sntp
– any (the word ” any” include all of them.)

You can also, specify a priority to order access

For exemple you can do this:

console(config-macl)#permit ip-source 192.168.99.1 service snmp
console(config-macl)#permit ip-source 192.168.100.10 service snmp
console(config-macl)#permit ip-source 192.168.99.1 service ssh
console(config-macl)#permit ip-source 192.168.100.10 service ssh
console(config-macl)# deny vlan 666 service any
console(config-macl)# deny ip-source 192.168.10.0 mask /24 service ssh
console(config-macl)# deny ip-source 192.168.200.0 mask /24 vlan 200 service snmp

After we do associate to the management access-class

console(config)#management access-class AdminList

Keep in minds:
If one of this service is not start, it’s not necessary to add in the management access-list

802.1x + Voice VLAN +MAB sur N-Series

On commence par déclarer en globale le serveur Radius (FreeRadius/PacketFence/Microsoft NPS/….)

authentication enable
authentication dynamic-vlan enable
dot1x system-auth-control
aaa authentication dot1x default radius
aaa authorization network default radius
radius server source-ip [IP du Vlan qui va joindre le serveur Radius]
radius server auth [IP.DU.SERVEUR.RADIUS]
name "RADIUS-SRVNAME1"
usage authmgr
key 0 "lacleduserveurradius"
exit
radius server acct [IP.DU.SERVEUR.RADIUS]
name "RADIUS-SRVNAME1"
exit

On active en globale le voice vlan, ainsi qu’un vlan de quarantaine si il venait a y avoir un problème d’authentification.

vlan 18
name "VLAN_TOIP"
exit
vlan 666
name "QUARANTAINE"
exit
switchport voice vlan

On configure le port pour permettre l’authentification 802.1x du client, sur échec on essaye un authentification par mac et sinon on drope dans le Vlan de Quarantaine.
Le téléphone identifiée via le LLDP, reçoit le VLAN Voix.
Cette configuration permet d’avoir l’interface “LAN” du téléphone qui reçoit une client qui va demander une authentification 802.1x.

LE MODE GENERAL EST OBLIGATOIRE.

interface Gi1/0/6
description "VOIP+PC"
switchport mode general
authentication host-mode multi-auth
authentication violation protect
authentication event fail action authorize vlan 666
authentication event no-response action authorize vlan 666
authentication periodic
dot1x timeout quiet-period 1
dot1x max-reauth-req 1
dot1x max-req 1
mab auth-type pap
authentication order dot1x mab
authentication priority dot1x mab
lldp tlv-select system-description system-capabilities management-address
lldp notification
lldp med confignotification
switchport voice vlan 16
switchport voice vlan override-authentication
exit


Config MAB sur N-Series

Dans une environnement ou on cherche à identifier l’ensemble des équipements se connectant au réseau, on active du 802.1x.
Pour des équipements n’ayant pas de possibilité de s’authentifier en 802.1x, comme des Imprimantes/Camera IP/IOT/etc.., il est généralement préconisé de mettre en place de l’authentification par @mac

Dans la config ci dessous en active l’allocation dynamique de VLAN et du MAB uniquement.

Si il est nécessaire de changer le format de la mac envoyé, il faut ajouter la commande suivante, a adapter selon les besoins.

mab request format attribute 1 groupsize 2 separator -

Les commandes suivantes, permettent l’activation des fonctions d’authentification 802.1x, l’allocation dynamique de Vlan et les paramètres de configuration du serveur Radius.

authentication enable
authentication dynamic-vlan enable
dot1x system-auth-control
aaa authentication dot1x default radius
aaa authorization network default radius

radius server key 7 "maybe"
radius server auth 192.168.100.23
primary
name "radius"
timeout 1
usage authmgr
exit
radius server acct 192.168.100.23
name "radius"
exit

Dans l’exemple, ci dessous seule l’authentification mab est activé sur le port. la sécurité est configuré pour bloqué le port en cas de pb d’authentification. Il y a une phase de re authentification periodique sur le port. 

interface Gi1/0/1
spanning-tree portfast
switchport mode general
authentication host-mode single-host
authentication max-users 1
authentication violation shutdown
authentication periodic
authentication timer reauthenticate 300
authentication order mab
authentication priority mab
authentication event fail retry 1
switchport port-security maximum 1
switchport port-security violation shutdown
exit

Par défaut, il est prévu 10 requette Radius par seconde, il est possible de modifier la valeur au Global

authentication critical recovery max-reauth 20

Suivant le serveur est parfois nécessaire de forcer le mode d’authentification, ici CHAP. Pour rappel cette configuration fonctionne avec du FreeRadius/PacketFence…

interface Gi1/0/1
mab auth-type chap
exit

Dans le cas d’une station sur laquelle vous souhaitez by-passer l’authentification 802.1x (mab ou autre)

interface Gi1/0/20
authentication port-control force-authorized
exit

Dans le cas des ports d’uplinks, là encore pas d’authentification possible, on filtre éventuellement les VLANs

interface Te1/0/1
switchport mode trunk
switchport trunk allowed vlan 10,20,30,40,50
authentication port-control force-authorized
exit

Utilisation Tacacs +

aaa authentication login "AuthSrv" tacacs local
aaa authentication enable "AuthSrvEna" tacacs line enable
aaa authorization exec "dfltExecAuthList" tacacs local none
aaa authorization commands "dfltCmdAuthList" tacacs none

tacacs-server host [IP.DU.SERVEUR.TACACS]
timeout 15
key 7 "La Clef Tacas"
exit


line console
exec-timeout 15
login authentication AuthSrv
exit

line ssh
exec-timeout 15
login authentication AuthSrv
enable authentication AuthSrvEna
exit

SI on positionne uniquement la ligne authorization sans le fallback (donc sans le none)

aaa authorization commands "dfltCmdAuthList" tacacs

Il faut rajouter le bloc “commands” ci dessus.

aaa authorization exec "dfltExecAuthList" tacacs local
aaa authorization commands "MyAuth" none
line ssh
authorization commands MyAuth

line console
authorization commands MyAuth

FreeRadius and Dnsmasq

I use DNSmasq to manage my local DHCP/DNS server, but unfortunatly there is no plugin’s or wrapper for FreeRadius. I have coded a quick and dirty Python program to do this:

  • Generate an authorize file
  • Generate and authorized_macs file (no more require with my settings)

My next step is use the “tag” with the “set:” section from dnsmasq for mark the vlan ID (for exemple set:vlan10).this code is also available on GitHub.

After launching the script, copy the authorize_conf file in the files folder in the freeradius 3.0 directory.

i use the $INCLUDE value in the default.conf for include it.

802.1x / MAB / N-Series

Fichier FreeRadius

C42C031E79AC    User-Password = "C42C031E79AC"
                Tunnel-Type = "VLAN",
                Tunnel-Medium-Type = "IEEE-802",
                Tunnel-Private-Group-ID = "10",
                Auth-Type := local
"demo"  Cleartext-Password := "demo"
                Tunnel-Type = "VLAN",
                Tunnel-Medium-Type = "IEEE-802",
                Tunnel-Private-Group-ID = "1",
                Auth-Type := local
d8a25e91ab06    User-Password = "d8a25e91ab06"

                Auth-Type := local
DEFAULT Auth-Type := Reject

        Reply-Message = "USERS.8021x - REJECT"


FRR Install from Github

The goal is install FRR under a Raspian Distro

Here, each step to download and install FRR.

Source: http://docs.frrouting.org/en/latest/installation.html

git clone https://github.com/FRRouting/frr.git

Here my script running througt a cron.

#!/bin/bash

#Changement de repository
cd /home/pi/gitclone/frr
# update du Git
git pull

# Build de l'Autoconf
./bootstrap.sh

#Processing du build
./configure --enable-exampledir=/usr/share/doc/frr/examples/ --localstatedir=/var/run/frr --sbindir=/usr/lib/frr --sysconfdir=/etc/frr --enable-vtysh --enable-isisd --enable-pimd --enable-watchfrr --enable-ospfclient=yes --enable-ospfapi=yes --enable-multipath=64 --enable-user=frr --enable-group=frr --enable-vty-group=frrvty --enable-configfile-mask=0640 --enable-logfile-mask=0640 --enable-rtadv --enable-fpm --enable-ldpd --with-pkg-git-version --with-pkg-extra-version=-MyOwn
FRRVersion

make
make check
sudo make install <<< *******