FreeNAS kernel python3.6 was killed: out of swap space

Ran into an error with FreeNAS (version 11.1-RELEASE) where the system was complaining about running out of swap space (first noticed there was a problem when getting repeated emails about ‘Unauthorized system reboot’). The error on the console and in /var/log/messages is:

servername kernel: pid 26118 (python3.6), uid 0, was killed: out of swap space

Fortunately NFS was still working, and console login (using the second console) worked. Used this to enable ssh:

service sshd start

and then I could login remotely which is more civilised. The problem was there was no swap:

# swapinfo
Device 1K-blocks Used Avail Capacity

Checking the disks showed there were 2 Gb swap partitions on all the data drives (default configuration):

# gpart show
=> 40 286749400 mfisyspd0 GPT (137G)
 40 1024 1 bios-boot (512K)
 1064 286748368 2 freebsd-zfs (137G)
 286749432 8 - free - (4.0K)

=> 40 286749400 mfisyspd1 GPT (137G)
 40 1024 1 bios-boot (512K)
 1064 286748368 2 freebsd-zfs (137G)
 286749432 8 - free - (4.0K)

=> 40 7814037088 mfisyspd2 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd3 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd4 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd5 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 234441568 nvd0 GPT (112G)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 230247168 2 freebsd-zfs (110G)
 234441600 8 - free - (4.0K)

=> 40 234441568 nvd1 GPT (112G)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 230247168 2 freebsd-zfs (110G)
 234441600 8 - free - (4.0K)

The swap could be remounted with

# swapon /dev/mfisyspd0p1

etc.

This works:

# swapinfo -h
Device 1K-blocks Used Avail Capacity
/dev/nvd0p1 2097152 516M 1.5G 25%
/dev/nvd1p1 2097152 77M 1.9G 4%
/dev/mfisyspd2p1 2097152 12M 2.0G 1%
/dev/mfisyspd3p1 2097152 11M 2.0G 1%
/dev/mfisyspd4p1 2097152 10M 2.0G 1%
/dev/mfisyspd5p1 2097152 10M 2.0G 1%
Total 12582912 636M 11G 5%

The question is – did these fall off at some point, or were they not mounted at boot (a couple of months ago?). Will experiment with a test system.