Category: BSD

Choosing preferred gateway in linux with two interfaces configured for dhcp

The situation:

  • Main network and private network (physically separate in this case)
  • Main network has dhcp server
  • Private network has dhcp server (server A, OPNsense in this case), which also acts as a router to main network
  • Want to set up a separate system (server B) that we can ssh, rdp etc. into from main network to access private network
    • Desire to use dhcp addressing for both interfaces on server B

A problem can arise with this setup if the system uses server A as it’s default gateway for whatever reason. The outside network only sees the route to server B via server A, and when you try to remote access server B from the main network (using the appropriate IP address for the main network interface) it fails, as it routes via server A and doesn’t go anywhere.

In theory this could be fixed by telling server A not to advertise itself as a gateway. In this case we want it to act as a gateway for everything on the private network except this one system, so we’d like to configure dhcpd on server A to exclude the option routers for this one case. For a static mapping entry on OPNsense the gateway option appears to imply you can type none here and it will unset the default gateway (in the same way that setting this at the main dhcp page for the interface does). This does not in fact happen – see the forum discussion here for example.

There are workarounds:

  • Edit the dhcpd conf file manually. Example
  • Unset the option for the server and then include it in all static entries (but this doesn’t cover dynamic entries)

Alternatively, we can change the behaviour of server B. As this is a one-off issue this may be preferable. We could just set up everything as static addresses, but there is an arguably more elegant way – use interface metrics.

In this case we have two Ethernet interfaces, both configured by /etc/network/interfaces and obtaining their addresses by dhcp. Both will have a metric of zero (running ip route shows this – if there’s no metric number then it is 0 by default). The default gateway is probably chosen by the first dhcp server to answer or some other semi-random condition. We can alter the metric of one (or both) interfaces to prioritise the one we want. If we have interfaces ens19 (main network) and ens18 (private network) for example, then an example is:

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

# Main network
allow-hotplug ens19
iface ens19 inet dhcp

# Private network
allow-hotplug ens18
iface ens18 inet dhcp
metric 100

Adding the metric 100 line to ens18 deprioritises it (as ens19 should get the default metric 0). Restarting the networking with systemctl restart networking and then running ip route gives something like

default via 172.40.204.1 dev ens19
default via 192.168.60.1 dev ens18 metric 100
172.40.204.0/24 dev ens19 proto kernel scope link src 172.40.204.27
192.168.60.0/24 dev ens18 proto kernel scope link src 192.168.60.8

Accessing server B from the main network using 172.40.204.27 should then work properly. As a bonus if ens19 looses it’s connection then it should fall back to the private network gateway (although for our particular use case this isn’t important).

Blocking or disabling autofs automounts with the -null map

Suppose you have a linux network setup with automounter maps that come from the network (via nis, sssd, LDAP etc.) and you want to block some of them acting on a particular system. In our case we have an automount map that acts on /opt and mounts various software packages from network shares. The problem with this is that you can’t then install your own stuff locally to /opt, which is what a lot of Debian/Ubuntu packages expect to be able to do.

It turns out there is a option in the automounter for this sort of situation. There is a built-in map called -null that blocks any further automounts to a particular mountpoint. In our case we want to block auto.opt, so we add a line to auto.master (somewhere before the bottom +auto.master line)

/opt  -null

Then restart the autofs service (if stuff was mounted on /opt then unmount it). Or reboot the system. You should find that you can put stuff in the local /opt.

To check the map is blocked you can also run

automount --dumpmaps

(also handy for checking what is actually meant to be mapped where).

Another way of doing this that leaves the system auto.master untouched is to create a file /etc/auto.master.d/opt.autofs (the first part of the name can be anything you want). Put the same contents in the file, e.g.

/opt  -null

Note that using this mechanism normally requires two files – one in /etc/auto.master.d/ and a map file that it refers to. In this case -null is a built-in map.

Unfortunately this option is not well documented. Places where it is referred to are:

There are also other built-in maps, e.g. -passwd, -hosts, -fedfs. Of these only the -hosts map is documented in the auto.master(5) man page.

-null is confirmed to work in CentOS 7, CentOS 8, Ubuntu 20.04, Debian 10.

TrueNAS and Windows clients – NTLMv2 issues

Situation – TrueNAS (or FreeNAS, or other Samba servers) serving a SMB share with NTLMv1 authentication disabled. A standalone Windows 10 system can connect to it, but a domain joined Win 10 system constantly claims wrong password.

The culprit here was a old group policy setting in the domain:

Network Security: LAN Manager authentication level

(found in Computer Configuration - Windows Settings - Security Settings - Local Policies - Security Options)

This was set to Send LM & NTLM - use NTLMv2 session security if negotiated, for backwards compatibility reasons with Win 2000 boxes and the like. This affects the registry key lmcompatibilitylevel (setting it to 1) under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa.

Unfortunately this is a bit misleading. According to this article:

Security Watch: The Most Misunderstood Windows Security Setting of All Time

This should negotiate better session security if possible, but does not actually send NTLMv2 requests or responses.

Thus trying to connect to a TrueNAS SMB share fails unless NTLMv1 Auth is explicitly enabled (in the service settings).

Ideally the group policy should be removed and the normal setting restored (NTLMv2 only). Or we can enable NTLMv1 on the share if it isn’t going to be a permanent setup.

Invalid self-signed certificates in FreeNAS 11

Had a problem with several FreeNAS 11 systems – when trying to enable https for the web interface I couldn’t get a working certificate.

  1. Generate internal CA.
  2. Generate internal certificate.
  3. Chrome complains the certificate is invalid and does not allow you to bypass it.
  4. Delete certificate and CA, repeat, same problem.

After mucking about with using upper and lower case for the certificate CN names etc, eventually got things to work by generating a second certificate with the same CA. Which makes me think it might be this problem:

FreeNAS-Generated Certificates: Buyer Beware

So either generating a couple of certificates (and using the second one) or changing the serial number of the CA to something bigger than 1 (and then generating a certificate) should work.

(Go to the CA tab, double-click the CA entry and scroll down to the bottom to view/edit the serial number).

Creating mirrored swap devices on FreeNAS 11

As per the previous post, I was having problems with the swap partitions on my main FreeNAS 11 server. All the swap partitions were there, but were not being used until manually mounted. It turns out that FreeNAS now has a new way of using these – it seems to set up mirror devices using gmirror and uses these for swap. There are a couple of gotchas here – the first is that if you do a gmirror list on a working system you will see:

Balance: prefer

and

Type: MANUAL

The command line to create the mirror is:

 # gmirror create -b prefer swap0 /dev/nvd0p1 /dev/nvd1p1

(in this case, I’m creating the first swap on my NVMe cache drives. Note that the standard 2 Gb partition 1 of type swap is already present on all the data drives).

Note that pretty much all the guides out there use gmirror label – this gives you a type: AUTOMATIC mirror. We don’t want this, presumably because FreeNAS has it’s own scanning system for these things.

Verify the mirror exists:

# gmirror list
Geom name: swap0
State: COMPLETE
Components: 2
Balance: prefer
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3774900941
Type: MANUAL
Providers:
1. Name: mirror/swap0
 Mediasize: 2147483648 (2.0G)
 Sectorsize: 512
 Mode: r0w0e0
Consumers:
1. Name: nvd0p1
 Mediasize: 2147483648 (2.0G)
 Sectorsize: 512
 Stripesize: 0
 Stripeoffset: 65536
 Mode: r1w1e1
 State: ACTIVE
 Priority: 0
 Flags: (null)
 GenID: 0
 SyncID: 1
 ID: 2013054055
2. Name: nvd1p1
 Mediasize: 2147483648 (2.0G)
 Sectorsize: 512
 Stripesize: 0
 Stripeoffset: 65536
 Mode: r1w1e1
 State: ACTIVE
 Priority: 1
 Flags: (null)
 GenID: 0
 SyncID: 1
 ID: 1984168260

Then test the detection logic:

midclt call disk.swaps_configure

This should return

["mirror/swap0"]

Looking at the logs, this also encrypts the swap, creating a device

/mirror/swap0.eli

It also appears to automount it, as it then appears when you do a swapinfo.

Note that the system log complains about the other swap devices I set up earlier, so it may be it’s only really happy with encrypted swap.

You can create more mirrors and run midclt call disk.swaps_configure again. After that swapinfo should show you all the details:

# swapinfo -h
Device 1K-blocks Used Avail Capacity
/dev/mirror/swap0.eli 2097152 632M 1.4G 31%
/dev/mirror/swap2.eli 2097152 25M 2.0G 1%
/dev/mirror/swap1.eli 2097152 25M 2.0G 1%
Total 6291456 682M 5.3G 11%

This setup should in theory automatically work on reboot. Unfortunately, as this hosts lots of VM disk images I don’t want to test this anytime soon…

FreeNAS kernel python3.6 was killed: out of swap space

Ran into an error with FreeNAS (version 11.1-RELEASE) where the system was complaining about running out of swap space (first noticed there was a problem when getting repeated emails about ‘Unauthorized system reboot’). The error on the console and in /var/log/messages is:

servername kernel: pid 26118 (python3.6), uid 0, was killed: out of swap space

Fortunately NFS was still working, and console login (using the second console) worked. Used this to enable ssh:

service sshd start

and then I could login remotely which is more civilised. The problem was there was no swap:

# swapinfo
Device 1K-blocks Used Avail Capacity

Checking the disks showed there were 2 Gb swap partitions on all the data drives (default configuration):

# gpart show
=> 40 286749400 mfisyspd0 GPT (137G)
 40 1024 1 bios-boot (512K)
 1064 286748368 2 freebsd-zfs (137G)
 286749432 8 - free - (4.0K)

=> 40 286749400 mfisyspd1 GPT (137G)
 40 1024 1 bios-boot (512K)
 1064 286748368 2 freebsd-zfs (137G)
 286749432 8 - free - (4.0K)

=> 40 7814037088 mfisyspd2 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd3 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd4 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd5 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 234441568 nvd0 GPT (112G)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 230247168 2 freebsd-zfs (110G)
 234441600 8 - free - (4.0K)

=> 40 234441568 nvd1 GPT (112G)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 230247168 2 freebsd-zfs (110G)
 234441600 8 - free - (4.0K)

The swap could be remounted with

# swapon /dev/mfisyspd0p1

etc.

This works:

# swapinfo -h
Device 1K-blocks Used Avail Capacity
/dev/nvd0p1 2097152 516M 1.5G 25%
/dev/nvd1p1 2097152 77M 1.9G 4%
/dev/mfisyspd2p1 2097152 12M 2.0G 1%
/dev/mfisyspd3p1 2097152 11M 2.0G 1%
/dev/mfisyspd4p1 2097152 10M 2.0G 1%
/dev/mfisyspd5p1 2097152 10M 2.0G 1%
Total 12582912 636M 11G 5%

The question is – did these fall off at some point, or were they not mounted at boot (a couple of months ago?). Will experiment with a test system.