Linux Mint: Watchdog configuration

List of tested devices

• The guide was tested on a systems with the specs listed below, but should be easily adaptable

Device: Airtop3
CPU: Intel(R) Xeon(R) E-2176G CPU @ 3.70GHz
BIOS: 10/28/2019 American Megatrends Inc. ATOP3.PRD.0.29.4
OS: Linux Mint 19.2 Tina
Kernel: 4.15.0-54-generic
Kernel module: wdat_wdt

Device: fitlet2
CPU: Intel(R) Atom(TM) Processor E3950
BIOS: 05/14/2018 American Megatrends Inc. FLT2.0.40.01.00
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: iTCO_wdt

Device: fitlet2
CPU: Intel(R) Atom(TM) Processor E3950
BIOS: 09/17/2018 American Megatrends Inc. FLT2.0.46.01.00
OS: Linux Mint 20 Ulyana
Kernel: 5.4.0-42-generic
Kernel module: iTCO_wdt

Device: fitlet1
CPU: AMD A10 Micro-6700T APU+AMD Radeon R6 Graphics
BIOS: 11/30/2015 American Megatrends Inc. SBCFLT_0.08.08
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: sp5100_tco

Device: IPC2
CPU: Intel(R) Core(TM) i5-4300U
BIOS: 05/30/2017 Compulab Ltd. IPC2_3.330.5
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: iTCO_wdt

Device: IPC3
CPU: Intel(R) Core(TM) i5-7200U
BIOS: 08/09/2018 American Megatrends Inc. 5.12_IPC3K.PRD.0.25.7
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: wdat_wdt

Notes

• fitlet1: kernel module i2c_piix4 should be blacklisted
  • Expand the below instructions

Add the following lines to the /etc/modprobe.d/blacklist.conf

# fitlet1: sp5100_tco conflicts with i2c_piix4 for I/O address 0x0cd6
blacklist i2c_piix4

Then, unload i2c_piix4 and update initramfs

$ sudo modprobe -r i2c_piix4
$ sudo update-initramfs -u

• IPC2-Haswell: iTCO Watchdog does not reset ACPI Watchdog timer
  • Expand the below instructions

• if the iTCO BIOS does not reset the ACPI Watchdog timer, you may need to disable the BIOS configuration from Linux

   BIOS --> South bridge --> Watchdog

Get super-user permissions:

   $ su
   Password: 

   $ apt install ioport

• Note: On Ubuntu versions below 18.04, ioport should be installed manually from:

   https://packages.ubuntu.com/bionic/ioport

And then run the following command:

   $ outl 0x1854 0x2000

To read it for validation please run (also as super-user):

   $ printf "0x%x\n" $(inl 0x1854)
   0x2000

To run this command automatically on each boot add it to file named /etc/rc.local. If you do not have such file, create it.

   #!/bin/sh -e
   #
   # rc.local
   #
   # This script is executed at the end of each multiuser runlevel.
   # Make sure that the script will "exit 0" on success or any other
   # value on error.
   #
   # In order to enable or disable this script just change the execution
   # bits.
   #
   # By default this script does nothing.
   outl 0x1854 0x2000
   exit 0

Make it executable

   $ sudo chmod +x /etc/rc.local

• Airtop3: Make sure you have correct BIOS settings:
  • Expand the below instructions

1. Press [ESC] on startup to enter BIOS settings
2. Chipset -> PCH-IO Configuration -> Enable TCO Timer is set to [Enabled]
3. If not, set to [Enabled]
4. Navigate to "Save & Exit" tab and choose Save changes and exit

• fitlet2: Make sure you have correct BIOS settings:
  • Expand the below instructions

1. Press [ESC] on startup to enter BIOS settings
2. At the bottom of the main screen check OS selection is set to [Linux]
3. If not, set to [Linux]
4. Navigate to "Save & Exit" tab and choose Save changes and exit.

Usage example

• The following usage example represented on fitlet2 with kernel module iTCO_wdt
  • For fitlet1 use kernel module sp5100_tco
  • For IPC3, Airtop3 use kernel module wdat_wdt

• Verify it can be loaded successfully and information about the device appears in dmesg output
• NOTE: kernel module wdat_wdt does not produce debug messages, dmesg output will be empty

$ sudo modprobe iTCO_wdt
$ dmesg | grep -i iTCO
[   34.741071] iTCO_vendor_support: vendor-support=0
[   34.742749] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   34.742824] iTCO_wdt: Found a Apollo Lake SoC TCO device (Version=5, TCOBASE=0x0460)
[   34.742993] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)

• Install watchdog program, it should run in the background automatically on each system start

$ sudo apt install watchdog

• Modify the following configuration files:

/etc/default/watchdog
change:                watchdog_module="none"
to:                    watchdog_module="iTCO_wdt"

/etc/watchdog.conf
uncomment line:        watchdog-device  = /dev/watchdog

• NOTE: bugfix required for Linux Mint 18.3

/lib/systemd/system/watchdog.service 
add "WantedBy=default.target" line under [Install] header, as per below:

[Install]
WantedBy=default.target

• Try watchdog service and ensure it is running correctly

$ sudo systemctl start watchdog
$ sudo systemctl status watchdog
● watchdog.service - watchdog daemon
  Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
  Active: active (running) since Tue 2018-04-17 03:30:58 EDT; 3min 3s ago
Main PID: 1143 (watchdog)
  CGroup: /system.slice/watchdog.service
          └─1143 /usr/sbin/watchdog

Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: int=1s realtime=yes sync=no soft=no mla=0 mem=0
Apr 17 03:30:58 denis-fitlet2 systemd[1]: Started watchdog daemon.
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: ping: no machine to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: file: no file to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: pidfile: no server process to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: interface: no interface to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: temperature: no sensors to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: watchdog now set to 60 seconds
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: hardware watchdog identity: iTCO_wdt

• Enable autostart of the watchdog service

$ sudo systemctl enable watchdog
Synchronizing state of watchdog.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable watchdog

• Reboot

• Verify the the driver was loaded automatically

$ dmesg | grep -i iTCO

• Simulate kernel panic (as root user):

$ su
Password:
$ echo 1 > /proc/sys/kernel/sysrq
$ echo c > /proc/sysrq-trigger

• As a result, the watchdog mechanism should reboot the system within 30 seconds.