Creating mirrored swap devices on FreeNAS 11

As per the previous post, I was having problems with the swap partitions on my main FreeNAS 11 server. All the swap partitions were there, but were not being used until manually mounted. It turns out that FreeNAS now has a new way of using these – it seems to set up mirror devices using gmirror and uses these for swap. There are a couple of gotchas here – the first is that if you do a gmirror list on a working system you will see:

Balance: prefer

and

Type: MANUAL

The command line to create the mirror is:

 # gmirror create -b prefer swap0 /dev/nvd0p1 /dev/nvd1p1

(in this case, I’m creating the first swap on my NVMe cache drives. Note that the standard 2 Gb partition 1 of type swap is already present on all the data drives).

Note that pretty much all the guides out there use gmirror label – this gives you a type: AUTOMATIC mirror. We don’t want this, presumably because FreeNAS has it’s own scanning system for these things.

Verify the mirror exists:

# gmirror list
Geom name: swap0
State: COMPLETE
Components: 2
Balance: prefer
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3774900941
Type: MANUAL
Providers:
1. Name: mirror/swap0
 Mediasize: 2147483648 (2.0G)
 Sectorsize: 512
 Mode: r0w0e0
Consumers:
1. Name: nvd0p1
 Mediasize: 2147483648 (2.0G)
 Sectorsize: 512
 Stripesize: 0
 Stripeoffset: 65536
 Mode: r1w1e1
 State: ACTIVE
 Priority: 0
 Flags: (null)
 GenID: 0
 SyncID: 1
 ID: 2013054055
2. Name: nvd1p1
 Mediasize: 2147483648 (2.0G)
 Sectorsize: 512
 Stripesize: 0
 Stripeoffset: 65536
 Mode: r1w1e1
 State: ACTIVE
 Priority: 1
 Flags: (null)
 GenID: 0
 SyncID: 1
 ID: 1984168260

Then test the detection logic:

midclt call disk.swaps_configure

This should return

["mirror/swap0"]

Looking at the logs, this also encrypts the swap, creating a device

/mirror/swap0.eli

It also appears to automount it, as it then appears when you do a swapinfo.

Note that the system log complains about the other swap devices I set up earlier, so it may be it’s only really happy with encrypted swap.

You can create more mirrors and run midclt call disk.swaps_configure again. After that swapinfo should show you all the details:

# swapinfo -h
Device 1K-blocks Used Avail Capacity
/dev/mirror/swap0.eli 2097152 632M 1.4G 31%
/dev/mirror/swap2.eli 2097152 25M 2.0G 1%
/dev/mirror/swap1.eli 2097152 25M 2.0G 1%
Total 6291456 682M 5.3G 11%

This setup should in theory automatically work on reboot. Unfortunately, as this hosts lots of VM disk images I don’t want to test this anytime soon…

FreeNAS kernel python3.6 was killed: out of swap space

Ran into an error with FreeNAS (version 11.1-RELEASE) where the system was complaining about running out of swap space (first noticed there was a problem when getting repeated emails about ‘Unauthorized system reboot’). The error on the console and in /var/log/messages is:

servername kernel: pid 26118 (python3.6), uid 0, was killed: out of swap space

Fortunately NFS was still working, and console login (using the second console) worked. Used this to enable ssh:

service sshd start

and then I could login remotely which is more civilised. The problem was there was no swap:

# swapinfo
Device 1K-blocks Used Avail Capacity

Checking the disks showed there were 2 Gb swap partitions on all the data drives (default configuration):

# gpart show
=> 40 286749400 mfisyspd0 GPT (137G)
 40 1024 1 bios-boot (512K)
 1064 286748368 2 freebsd-zfs (137G)
 286749432 8 - free - (4.0K)

=> 40 286749400 mfisyspd1 GPT (137G)
 40 1024 1 bios-boot (512K)
 1064 286748368 2 freebsd-zfs (137G)
 286749432 8 - free - (4.0K)

=> 40 7814037088 mfisyspd2 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd3 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd4 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 7814037088 mfisyspd5 GPT (3.6T)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 7809842688 2 freebsd-zfs (3.6T)
 7814037120 8 - free - (4.0K)

=> 40 234441568 nvd0 GPT (112G)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 230247168 2 freebsd-zfs (110G)
 234441600 8 - free - (4.0K)

=> 40 234441568 nvd1 GPT (112G)
 40 88 - free - (44K)
 128 4194304 1 freebsd-swap (2.0G)
 4194432 230247168 2 freebsd-zfs (110G)
 234441600 8 - free - (4.0K)

The swap could be remounted with

# swapon /dev/mfisyspd0p1

etc.

This works:

# swapinfo -h
Device 1K-blocks Used Avail Capacity
/dev/nvd0p1 2097152 516M 1.5G 25%
/dev/nvd1p1 2097152 77M 1.9G 4%
/dev/mfisyspd2p1 2097152 12M 2.0G 1%
/dev/mfisyspd3p1 2097152 11M 2.0G 1%
/dev/mfisyspd4p1 2097152 10M 2.0G 1%
/dev/mfisyspd5p1 2097152 10M 2.0G 1%
Total 12582912 636M 11G 5%

The question is – did these fall off at some point, or were they not mounted at boot (a couple of months ago?). Will experiment with a test system.

Setting proxy in Windows Server 2016 for the Update service

In Windows Server 2016 the usual trick of just setting a proxy server in IE doesn’t seem to work. The proxy can be set from the command line, but there is some contradictory advice out there. What worked for me was:

netsh winhttp set proxy proxy-server="your.proxy.server:port" bypass-list="*.your.local.domain"

This should result in:

Current WinHTTP proxy settings:

    Proxy Server(s) : your.proxy.server:port
    Bypass List : *.your.local.domain

You can also check this with

netsh winhttp show proxy

You are meant to be able to import this from IE, but that gave me a syntax error.

You may need to restart the update service after this to kick it into life.

Connecting to NFS shares at boot using fstab in Debian 9 Stretch

Note – this fix in principle should work on most systemd distributions.

Problem – trying to get a Debian 9 system to mount an NFS share at boot. This was declared in /etc/fstab in the normal way, but kept failing on boot. However, once the system was up you could log in and do a mount -a, which would work fine. Reading around, it looks like a case of the system trying to mount before the network is up (and in this case the network should be reliable, as it’s an internal one between a VM and it’s host…)

Tried using the bg option first, which should mount in the background and come up eventually, but still got error on boot.

192.168.45.1:/export  /hostshare  nfs  bg,rw,soft  0   0

There is another option that in theory should help: _netdev. I haven’t tried this yet.

What does work is adding an option x-systemd.automount. This, unsurprisingly, tells systemd to try and mount the share on demand. So changing the line in fstab to read:

192.168.45.1:/export  /hostshare  nfs  rw,soft,x-systemd.automount  0   0

works. Booting the system gives no errors (on the console anyway). The share does not show as mounted until the local mountpoint is accessed, and then it works without complaint.

Footnotes

  1. The context for this is a VM running under VirtualBox. The VM is Debian 9, the host is Ubuntu 17.10. The VM has one network interface with the default NAT setup to talk to the outside world, and a second interface to talk to a host-only network. This allows you to SSH into the the guest from the host, and also allows this NFS setup. You can use the VirtualBox shared folder setup to transfer files, but I figured as both the host and guest were Linux NFS would be easier (and not require the Guest Extensions to be installed on the guest).
  2. Debian 9 wouldn’t successfully install on the laptop, but I needed it for an easy install of LALSuite (Debian is a reference system for this, Ubuntu isn’t and has dependency issues). Hence this rather complicated setup. Fortunately LALSuite is entirely command line based…
  3. Yes, Docker or similar would be more efficient. I’m not so familiar with it and it’s a bit of a pain to get it talking to the host filesystem. I’d argue that running a full VM is slightly more portable, although you’d need to change how the filesharing is set up on a Windows or Mac host.

Forcing Windows 10 Enterprise to install on a system with a Home version key in the BIOS

Problem – I had a system (HP Spectre) with Windows 10 Home edition which I wanted to upgrade to Enterprise (to get BitLocker). The trouble is, even when using the Windows 10 Enterprise install ISO and a clean install it installed Home. This is because it detects a Home key embedded in the BIOS and automatically installs it without giving you a choice.

You can apparently force it to install the version you want by including EI.cfg and PID.txt files in the install ISO (or USB key). I had a look at our ISO, which does include EI.cfg. The contents are:

[EditionID]
Enterprise

[Channel]
Volume

[VL]
1

Interestingly, according to the Microsoft documentation this is wrong – [Channel] should be either Retail or OEM…

There is no PID.txt file. According to some people this is also required. I was going to try this, but then found a simpler solution. You can run setup.exe from the DVD (assuming you have a functioning install already, like I had here) and include command line switches. Including specifying the product key.

setup.exe /PKey NPPR9-FWDCX-D2C8J-H872K-2YT43

(List of KMS client product keys)

Hey presto, this time the upgrade wizard only gives you the option of a clean(ish) install (as Win 10 Home to Enterprise is not a supported in-place upgrade for some reason), and the summary explicitly confirms you are upgrading to Enterprise.

And then you wait ages…

Note that if you have a system with Pro, then in theory you can change the product key (there are several ways to do this) to get to Enterprise. On the other hand, Pro can join domains and has BitLocker anyway.

Monitoring GPU temperatures with nvidia-smi and Check MK (OMD)

The Nvidia monitoring setup described at https://elwe.rhrk.uni-kl.de/howto/ worked in Check MK 1.2.8, but fails in 1.4. After some modification things now work – it required some modification of the check script /omd/yoursite/local/share/check_mk/checks/nvidia_smi. The two modifications needed were:

Remove the grouping of nvidia_smi.errors1 and 2 (I can live with this as our GTX1070 doesn’t report this anyway).

Remove the unicode degree characters from the temperature output, as this seems to cause the system to choke on the textual output.

Needed to delete and recreate the host to get it to work properly – possibly unicode characters hanging around in the generated graph definitions or similar?

#!/usr/bin/python
# -*- encoding: utf-8; py-indent-offset: 4 -*-
# +------------------------------------------------------------------+
# | ____ _ _ __ __ _ __ |
# | / ___| |__ ___ ___| | __ | \/ | |/ / |
# | | | | '_ \ / _ \/ __| |/ / | |\/| | ' / |
# | | |___| | | | __/ (__| < | | | | . \ | # | \____|_| |_|\___|\___|_|\_\___|_| |_|_|\_\ | # | | # | Copyright Mathias Kettner 2012 mk@mathias-kettner.de | # +------------------------------------------------------------------+ # # This file is part of Check_MK. # The official homepage is at http://mathias-kettner.de/check_mk. # # check_mk is free software; you can redistribute it and/or modify it # under the terms of the GNU General Public License as published by # the Free Software Foundation in version 2. check_mk is distributed # in the hope that it will be useful, but WITHOUT ANY WARRANTY; with- # out even the implied warranty of MERCHANTABILITY or FITNESS FOR A # PARTICULAR PURPOSE. See the GNU General Public License for more de- # ails. You should have received a copy of the GNU General Public # License along with GNU Make; see the file COPYING. If not, write # to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, # Boston, MA 02110-1301 USA. ####################################### # Check developed by ####################################### # Dr. Markus Hillenbrand # University of Kaiserslautern, Germany # hillenbr@rhrk.uni-kl.de ####################################### # Tweaked by Jamie Scott # University of Glasgow # Jamie.Scott@glasgow.ac.uk ####################################### # the inventory functions def inventory_nvidia_smi_fan(info): inventory = [] for line in info: if line[2] != 'N/A': inventory.append( ("GPU"+line[0], "", None) ) return inventory def inventory_nvidia_smi_gpuutil(info): inventory = [] for line in info: if line[3] != 'N/A': inventory.append( ("GPU"+line[0], "", None) ) return inventory def inventory_nvidia_smi_memutil(info): inventory = [] for line in info: if line[4] != 'N/A': inventory.append( ("GPU"+line[0], "", None) ) return inventory def inventory_nvidia_smi_errors1(info): inventory = [] for line in info: if line[5] != 'N/A': inventory.append( ("GPU"+line[0], "", None) ) return inventory def inventory_nvidia_smi_errors2(info): inventory = [] for line in info: if line[6] != 'N/A': inventory.append( ("GPU"+line[0], "", None) ) return inventory def inventory_nvidia_smi_temp(info): inventory = [] for line in info: if line[7] != 'N/A': inventory.append( ("GPU"+line[0], "", None) ) return inventory def inventory_nvidia_smi_power(info): inventory = [] for line in info: if line[8] != 'N/A' and line[9] != "N/A": inventory.append( ("GPU"+line[0], "", None) ) return inventory # the check functions def check_nvidia_smi_fan(item, params, info): for line in info: if "GPU"+line[0] == item: value = int(line[2]) perfdata = [('fan', value, 90, 95, 0, 100 )] if value > 95:
return (2, "CRITICAL - %s fan speed is %d%%" % (line[1], value), perfdata)
elif value > 90:
return (1, "WARNING - %s fan speed is %d%%" % (line[1], value), perfdata)
else:
return (0, "OK - %s fan speed is %d%%" % (line[1], value), perfdata)
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

def check_nvidia_smi_gpuutil(item, params, info):
for line in info:
if "GPU"+line[0] == item:
value = int(line[3])
perfdata = [('gpuutil', value, 100, 100, 0, 100 )]
return (0, "OK - %s utilization is %s%%" % (line[1], value), perfdata)
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

def check_nvidia_smi_memutil(item, params, info):
for line in info:
if "GPU"+line[0] == item:
value = int(line[4])
perfdata = [('memutil', value, 100, 100, 0, 100 )]
if value > 95:
return (2, "CRITICAL - %s memory utilization is %d%%" % (line[1], value), perfdata)
elif value > 90:
return (1, "WARNING - %s memory utilization is %d%%" % (line[1], value), perfdata)
else:
return (0, "OK - %s memory utilization is %d%%" % (line[1], value), perfdata)
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

def check_nvidia_smi_errors1(item, params, info):
for line in info:
if "GPU"+line[0] == item:
value = int(line[5])
if value > 500:
return (2, "CRITICAL - %s single bit error counter is %d" % (line[1], value))
if value > 100:
return (1, "WARNING - %s single bit error counter is %d" % (line[1], value))
else:
return (0, "OK - %s single bit error counter is %d" % (line[1], value))
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

def check_nvidia_smi_errors2(item, params, info):
for line in info:
if "GPU"+line[0] == item:
value = int(line[6])
if value > 500:
return (2, "CRITICAL - %s double bit error counter is %d" % (line[1], value))
if value > 100:
return (1, "WARNING - %s double bit error counter is %d" % (line[1], value))
else:
return (0, "OK - %s double bit error counter is %d" % (line[1], value))
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

def check_nvidia_smi_temp(item, params, info):
for line in info:
if "GPU"+line[0] == item:
value = int(line[7])
perfdata = [('temp', value, 80, 90, 0, 95 )]
if value > 90:
return (2, "CRITICAL - %s temperature is %dC" % (line[1], value), perfdata)
elif value > 80:
return (1, "WARNING - %s temperature is %dC" % (line[1], value), perfdata)
else:
return (0, "OK - %s temperature is %dC" % (line[1], value), perfdata)
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

def check_nvidia_smi_power(item, params, info):
for line in info:
if "GPU"+line[0] == item:
draw = float(line[8])
limit = float(line[9])
value = draw * 100.0 / limit
perfdata = [('power', draw, limit * 0.8, limit * 0.9, 0, limit )]
if value > 90:
return (2, "CRITICAL - %s power utilization is %d%% of %dW" % (line[1], value, limit), perfdata)
elif value > 80:
return (1, "WARNING - %s power utilization is %d%% of %dW" % (line[1], value, limit), perfdata)
else:
return (0, "OK - %s power utilization is %d%% of %dW" % (line[1], value, limit), perfdata)
return (3, "UNKNOWN - GPU %s not found in agent output" % item)

# declare the check to Check_MK

check_info['nvidia_smi.fan'] = (check_nvidia_smi_fan, "%s fan speed" , 1, inventory_nvidia_smi_fan)
check_info['nvidia_smi.gpuutil'] = (check_nvidia_smi_gpuutil, "%s utilization" , 1, inventory_nvidia_smi_gpuutil)
check_info['nvidia_smi.memutil'] = (check_nvidia_smi_memutil, "%s memory" , 1, inventory_nvidia_smi_memutil)
#check_info['nvidia_smi.errors1'] = (check_nvidia_smi_errors1, "%s errors single" , 0, inventory_nvidia_smi_errors1)
#check_info['nvidia_smi.errors2'] = (check_nvidia_smi_errors2, "%s errors double" , 0, inventory_nvidia_smi_errors2)
check_info['nvidia_smi.temp'] = (check_nvidia_smi_temp, "%s temperature" , 1, inventory_nvidia_smi_temp)
check_info['nvidia_smi.power'] = (check_nvidia_smi_power, "%s power" , 1, inventory_nvidia_smi_power)

#checkgroup_of['nvidia_smi.errors1'] = 'hw_errors'
#checkgroup_of['nvidia_smi.errors2'] = 'hw_errors'

ResourceSpace cron and database notes

Ran into a couple of issues today:

Note: system setup is Debian 9 with standard options (Apache 2.4, PHP 7.0, MariaDB 10.1)

Cron

The documentation implies you should run cron_copy_hitcount.php as a cron job. However, the new correct way seems to be to run batch/cron.php, which runs a bunch of sub-jobs. I’ve got this set up in cron.daily as:

#!/bin/sh
wget -q -r http://localhost/resourcespace/batch/cron.php

We’ll see if this works. Certainly running it directly by browsing to it seems to work.

LDAP

Trying to activate the simpleldap plugin threw up two problems:

php-ldap wasn’t installed – easy enough. Note apache needs a restart after installing…

Second error was a problem with the database – the plugin couldn’t create a table, with error

Specified key was too long; max key length is 767 bytes

This seems to be because when I created the database the character set used was utf8mb4_general_ci, which in the worst case uses 4 bytes per character. If you try to create a index key with 255 characters you run into this limit.

The solution was to change the database to use utf8_general_ci. This allowed the plugin to create the simpleldap_groupmap table with utf8_general_ci. The rest of the database is still utf8mb4_general_ci, but as it has been created already without an issue we should be ok.