Last modified on 28 July 2020, at 07:00

Difference between revisions of "Linux Mint: Watchdog configuration"

(List of tested devices)   (change visibility)
 
(77 intermediate revisions by 2 users not shown)
Line 1: Line 1:
1. Find out watchdog kernel module name for your platform, possibly using using google
+
== List of tested devices ==
*For all Compulab Intel products, it should be named - '''"iTCO_wdt"'''
+
* The guide was tested on a systems with the specs listed below, but should be easily adaptable
 +
Device: Airtop3
 +
CPU: Intel(R) Xeon(R) E-2176G CPU @ 3.70GHz
 +
BIOS: 10/28/2019 American Megatrends Inc. ATOP3.PRD.0.29.4
 +
OS: Linux Mint 19.2 Tina
 +
Kernel: 4.15.0-54-generic
 +
Kernel module: wdat_wdt
  
2. Verify it can be loaded successfully information on the device appears in dmesg output
+
Device: fitlet2
 +
CPU: Intel(R) Atom(TM) Processor E3950
 +
BIOS: 05/14/2018 American Megatrends Inc. FLT2.0.40.01.00
 +
OS: Linux Mint 19 Tara
 +
Kernel: 4.15.0-23-generic
 +
Kernel module: iTCO_wdt
  
'''$ sudo modprobe iTCO_wdt'''
+
Device: fitlet2
 +
CPU: Intel(R) Atom(TM) Processor E3950
 +
BIOS: 09/17/2018 American Megatrends Inc. FLT2.0.46.01.00
 +
OS: Linux Mint 20 Ulyana
 +
Kernel: 5.4.0-42-generic
 +
Kernel module: iTCO_wdt
  
'''$ dmesg | tail'''
+
Device: fitlet1
:output:
+
CPU: AMD A10 Micro-6700T APU+AMD Radeon R6 Graphics
    iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
+
BIOS: 11/30/2015 American Megatrends Inc. SBCFLT_0.08.08
    iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)
+
OS: Linux Mint 19 Tara
    iTCO_wdt: initialized. heartbeat=60 sec (nowayout=0)
+
Kernel: 4.15.0-23-generic
 +
Kernel module: sp5100_tco
  
3. Install watchdog program, it should run in the background automatically on each system start
+
Device: IPC2
:'''$ sudo apt install watchdog'''
+
CPU: Intel(R) Core(TM) i5-4300U
 +
BIOS: 05/30/2017 Compulab Ltd. IPC2_3.330.5
 +
OS: Linux Mint 19 Tara
 +
Kernel: 4.15.0-23-generic
 +
Kernel module: iTCO_wdt
  
4. Modify the following configuration files:
+
Device: IPC3
 +
CPU: Intel(R) Core(TM) i5-7200U
 +
BIOS: 08/09/2018 American Megatrends Inc. 5.12_IPC3K.PRD.0.25.7
 +
OS: Linux Mint 19 Tara
 +
Kernel: 4.15.0-23-generic
 +
Kernel module: wdat_wdt
 +
<br>
 +
 
 +
== Notes ==
 +
*'''fitlet1''': kernel module i2c_piix4 should be blacklisted
 +
**Expand the below instructions
 +
<div class="toccolours mw-collapsible mw-collapsed" overflow:auto;">
 +
Add the following lines to the /etc/modprobe.d/blacklist.conf
 +
# fitlet1: sp5100_tco conflicts with i2c_piix4 for I/O address 0x0cd6
 +
blacklist i2c_piix4
 +
Then, unload i2c_piix4 and update initramfs
 +
$ sudo modprobe -r i2c_piix4
 +
$ sudo update-initramfs -u
 +
</div>
 +
*'''IPC2-Haswell''': iTCO Watchdog does not reset ACPI Watchdog timer
 +
**Expand the below instructions
 +
<div class="toccolours mw-collapsible mw-collapsed" overflow:auto;">
 +
* if the iTCO BIOS does not reset the ACPI Watchdog timer, you may need to disable the BIOS configuration from Linux
 +
    BIOS --> South bridge --> Watchdog
 +
 
 +
Get super-user permissions:
 +
    $ su
 +
    Password:
 +
 
 +
    $ apt install ioport
 +
* '''Note:''' On Ubuntu versions below 18.04, ioport should be installed manually from:
 +
    https://packages.ubuntu.com/bionic/ioport
 +
 
 +
 
 +
And then run the following command:
 +
    $ outl 0x1854 0x2000
 +
 
 +
To read it for validation please run (also as super-user):
 +
    $ printf "0x%x\n" $(inl 0x1854)
 +
    0x2000
 +
 
 +
To run this command automatically on each boot add it to file named /etc/rc.local.
 +
If you do not have such file, create it.
 +
 
 +
    #!/bin/sh -e
 +
    #
 +
    # rc.local
 +
    #
 +
    # This script is executed at the end of each multiuser runlevel.
 +
    # Make sure that the script will "exit 0" on success or any other
 +
    # value on error.
 +
    #
 +
    # In order to enable or disable this script just change the execution
 +
    # bits.
 +
    #
 +
    # By default this script does nothing.
 +
    outl 0x1854 0x2000
 +
    exit 0
 +
 
 +
Make it executable
 +
 
 +
    $ sudo chmod +x /etc/rc.local
 +
</div>
 +
*'''Airtop3''': Make sure you have correct BIOS settings:
 +
**Expand the below instructions
 +
<div class="toccolours mw-collapsible mw-collapsed">
 +
# Press '''[ESC]''' on startup to enter BIOS settings
 +
# Chipset -> PCH-IO Configuration -> Enable TCO Timer is set to '''[Enabled]'''
 +
# If not, set to '''[Enabled]'''
 +
# Navigate to '''"Save & Exit"''' tab and choose '''Save changes and exit'''
 +
</div>
 +
 
 +
*'''fitlet2''': Make sure you have correct BIOS settings:
 +
**Expand the below instructions
 +
<div class="toccolours mw-collapsible mw-collapsed">
 +
# Press '''[ESC]''' on startup to enter BIOS settings
 +
# At the bottom of the main screen check '''OS selection''' is set to '''[Linux]'''
 +
# If not, set to '''[Linux]'''
 +
# Navigate to '''"Save & Exit"''' tab and choose '''Save changes and exit'''.
 +
</div>
 +
 
 +
<br>
 +
 
 +
== Usage example ==
 +
 
 +
* The following usage example represented on fitlet2 with kernel module iTCO_wdt
 +
** For fitlet1 use kernel module sp5100_tco
 +
** For IPC3, Airtop3 use kernel module wdat_wdt
 +
<br>
 +
* Verify it can be loaded successfully and information about the device appears in dmesg output
 +
* NOTE: kernel module wdat_wdt does not produce debug messages, dmesg output will be empty
 +
 
 +
$ sudo modprobe iTCO_wdt
 +
$ dmesg | grep -i iTCO
 +
[  34.741071] iTCO_vendor_support: vendor-support=0
 +
[  34.742749] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
 +
[  34.742824] iTCO_wdt: Found a Apollo Lake SoC TCO device (Version=5, TCOBASE=0x0460)
 +
[  34.742993] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
 +
 
 +
* Install watchdog program, it should run in the background automatically on each system start
 +
$ sudo apt install watchdog
 +
 
 +
* Modify the following configuration files:
 +
/etc/default/watchdog
 +
change:                watchdog_module="none"
 +
to:                    watchdog_module="iTCO_wdt"
 +
 
 +
/etc/watchdog.conf
 +
uncomment line:        watchdog-device  = /dev/watchdog
 +
 
 +
* NOTE: bugfix required for Linux Mint 18.3
 +
/lib/systemd/system/watchdog.service
 +
add "WantedBy=default.target" line under [Install] header, as per below:
 
   
 
   
:'''/etc/watchdog.conf'''
+
[Install]
    uncomment line:        watchdog-device = /dev/watchdog
+
  WantedBy=default.target
 
   
 
   
:'''/etc/default/watchdog'''
+
 
    change:               watchdog_module="none"
+
* Try watchdog service and ensure it is running correctly
    to:                   watchdog_module=" iTCO_wdt"
+
$ sudo systemctl start watchdog
 +
$ sudo systemctl status watchdog
 +
● watchdog.service - watchdog daemon
 +
  Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
 +
  Active: active (running) since Tue 2018-04-17 03:30:58 EDT; 3min 3s ago
 +
Main PID: 1143 (watchdog)
 +
  CGroup: /system.slice/watchdog.service
 +
          └─1143 /usr/sbin/watchdog
 
   
 
   
:'''/lib/systemd/system/watchdog.service'''
+
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: int=1s realtime=yes sync=no soft=no mla=0 mem=0
::add "WantedBy=default.target" line under [Install] header, as per below:
+
Apr 17 03:30:58 denis-fitlet2 systemd[1]: Started watchdog daemon.
   
+
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: ping: no machine to check
  [Install]
+
  Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: file: no file to check
  WantedBy=default.target
+
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: pidfile: no server process to check
   
+
  Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: interface: no interface to check
* NOTE: this a bugfix that required for LinuxMint18.3.
+
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: temperature: no sensors to check
   
+
  Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no
5. Reboot
+
  Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: watchdog now set to 60 seconds
   
+
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: hardware watchdog identity: iTCO_wdt
6. Verify the the driver is loaded automatically:
+
  
:'''$ dmesg | grep wdt'''
+
* Enable autostart of the watchdog service
 +
$ sudo systemctl enable watchdog
 +
Synchronizing state of watchdog.service with SysV init with /lib/systemd/systemd-sysv-install...
 +
Executing /lib/systemd/systemd-sysv-install enable watchdog
 +
 
 +
* Reboot
 
   
 
   
7. Simulate kernel panic (as root user):
+
* Verify the the driver was loaded automatically
    $ su
+
$ dmesg | grep -i iTCO
    Password:
+
    $ echo c > /proc/sysrq-trigger
+
* Simulate kernel panic (as root user):
 +
$ su
 +
Password:
 +
$ echo 1 > /proc/sys/kernel/sysrq
 +
$ echo c > /proc/sysrq-trigger
  
*As a result, the watchdog mechanism should reboot the system within 60 seconds.
+
* As a result, the watchdog mechanism should reboot the system within 30 seconds.

Latest revision as of 07:00, 28 July 2020

List of tested devices

  • The guide was tested on a systems with the specs listed below, but should be easily adaptable
Device: Airtop3
CPU: Intel(R) Xeon(R) E-2176G CPU @ 3.70GHz
BIOS: 10/28/2019 American Megatrends Inc. ATOP3.PRD.0.29.4
OS: Linux Mint 19.2 Tina
Kernel: 4.15.0-54-generic
Kernel module: wdat_wdt
Device: fitlet2
CPU: Intel(R) Atom(TM) Processor E3950
BIOS: 05/14/2018 American Megatrends Inc. FLT2.0.40.01.00
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: iTCO_wdt
Device: fitlet2
CPU: Intel(R) Atom(TM) Processor E3950
BIOS: 09/17/2018 American Megatrends Inc. FLT2.0.46.01.00
OS: Linux Mint 20 Ulyana
Kernel: 5.4.0-42-generic
Kernel module: iTCO_wdt
Device: fitlet1
CPU: AMD A10 Micro-6700T APU+AMD Radeon R6 Graphics
BIOS: 11/30/2015 American Megatrends Inc. SBCFLT_0.08.08
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: sp5100_tco
Device: IPC2
CPU: Intel(R) Core(TM) i5-4300U
BIOS: 05/30/2017 Compulab Ltd. IPC2_3.330.5
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: iTCO_wdt
Device: IPC3
CPU: Intel(R) Core(TM) i5-7200U
BIOS: 08/09/2018 American Megatrends Inc. 5.12_IPC3K.PRD.0.25.7
OS: Linux Mint 19 Tara
Kernel: 4.15.0-23-generic
Kernel module: wdat_wdt


Notes

  • fitlet1: kernel module i2c_piix4 should be blacklisted
    • Expand the below instructions

Add the following lines to the /etc/modprobe.d/blacklist.conf

# fitlet1: sp5100_tco conflicts with i2c_piix4 for I/O address 0x0cd6
blacklist i2c_piix4

Then, unload i2c_piix4 and update initramfs

$ sudo modprobe -r i2c_piix4
$ sudo update-initramfs -u
  • IPC2-Haswell: iTCO Watchdog does not reset ACPI Watchdog timer
    • Expand the below instructions
  • if the iTCO BIOS does not reset the ACPI Watchdog timer, you may need to disable the BIOS configuration from Linux
   BIOS --> South bridge --> Watchdog

Get super-user permissions:

   $ su
   Password: 
   $ apt install ioport
  • Note: On Ubuntu versions below 18.04, ioport should be installed manually from:
   https://packages.ubuntu.com/bionic/ioport


And then run the following command:

   $ outl 0x1854 0x2000

To read it for validation please run (also as super-user):

   $ printf "0x%x\n" $(inl 0x1854)
   0x2000

To run this command automatically on each boot add it to file named /etc/rc.local. If you do not have such file, create it.

   #!/bin/sh -e
   #
   # rc.local
   #
   # This script is executed at the end of each multiuser runlevel.
   # Make sure that the script will "exit 0" on success or any other
   # value on error.
   #
   # In order to enable or disable this script just change the execution
   # bits.
   #
   # By default this script does nothing.
   outl 0x1854 0x2000
   exit 0

Make it executable

   $ sudo chmod +x /etc/rc.local
  • Airtop3: Make sure you have correct BIOS settings:
    • Expand the below instructions
  1. Press [ESC] on startup to enter BIOS settings
  2. Chipset -> PCH-IO Configuration -> Enable TCO Timer is set to [Enabled]
  3. If not, set to [Enabled]
  4. Navigate to "Save & Exit" tab and choose Save changes and exit
  • fitlet2: Make sure you have correct BIOS settings:
    • Expand the below instructions
  1. Press [ESC] on startup to enter BIOS settings
  2. At the bottom of the main screen check OS selection is set to [Linux]
  3. If not, set to [Linux]
  4. Navigate to "Save & Exit" tab and choose Save changes and exit.


Usage example

  • The following usage example represented on fitlet2 with kernel module iTCO_wdt
    • For fitlet1 use kernel module sp5100_tco
    • For IPC3, Airtop3 use kernel module wdat_wdt


  • Verify it can be loaded successfully and information about the device appears in dmesg output
  • NOTE: kernel module wdat_wdt does not produce debug messages, dmesg output will be empty
$ sudo modprobe iTCO_wdt
$ dmesg | grep -i iTCO
[   34.741071] iTCO_vendor_support: vendor-support=0
[   34.742749] iTCO_wdt: Intel TCO WatchDog Timer Driver v1.11
[   34.742824] iTCO_wdt: Found a Apollo Lake SoC TCO device (Version=5, TCOBASE=0x0460)
[   34.742993] iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
  • Install watchdog program, it should run in the background automatically on each system start
$ sudo apt install watchdog
  • Modify the following configuration files:
/etc/default/watchdog
change:                watchdog_module="none"
to:                    watchdog_module="iTCO_wdt"
/etc/watchdog.conf
uncomment line:        watchdog-device  = /dev/watchdog
  • NOTE: bugfix required for Linux Mint 18.3
/lib/systemd/system/watchdog.service 
add "WantedBy=default.target" line under [Install] header, as per below:

[Install]
WantedBy=default.target

  • Try watchdog service and ensure it is running correctly
$ sudo systemctl start watchdog
$ sudo systemctl status watchdog
● watchdog.service - watchdog daemon
  Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
  Active: active (running) since Tue 2018-04-17 03:30:58 EDT; 3min 3s ago
Main PID: 1143 (watchdog)
  CGroup: /system.slice/watchdog.service
          └─1143 /usr/sbin/watchdog

Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: int=1s realtime=yes sync=no soft=no mla=0 mem=0
Apr 17 03:30:58 denis-fitlet2 systemd[1]: Started watchdog daemon.
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: ping: no machine to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: file: no file to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: pidfile: no server process to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: interface: no interface to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: temperature: no sensors to check
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: watchdog now set to 60 seconds
Apr 17 03:30:58 denis-fitlet2 watchdog[1143]: hardware watchdog identity: iTCO_wdt
  • Enable autostart of the watchdog service
$ sudo systemctl enable watchdog
Synchronizing state of watchdog.service with SysV init with /lib/systemd/systemd-sysv-install...
Executing /lib/systemd/systemd-sysv-install enable watchdog
  • Reboot
  • Verify the the driver was loaded automatically
$ dmesg | grep -i iTCO

  • Simulate kernel panic (as root user):
$ su
Password:
$ echo 1 > /proc/sys/kernel/sysrq
$ echo c > /proc/sysrq-trigger
  • As a result, the watchdog mechanism should reboot the system within 30 seconds.