SMART Input returning exit status 2 for active drives but not when run manually from inside the container #17370
Description
Relevant telegraf.conf
# Prometheus output
[[outputs.prometheus_client]]
listen = ":9273"
# SMART monitoring
[[inputs.smart]]
path_smartctl = "/usr/sbin/smartctl"
nocheck = "standby"
attributes = true
devices = ["/dev/sdb", "/dev/sde", "/dev/sdc"]Logs from Telegraf
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/932511d76b14"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/021e49962264"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/0324553c4a51"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/4b5c1f380496"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8a2b3b6d6f06"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/dc93de697727"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/272ed0de122f"): no such file or directory
2025-07-21T19:31:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/556ea498fb7b"): no such file or directory
2025-07-21T19:31:06Z D! [outputs.prometheus_client] Wrote batch of 131 metrics in 8.860129ms
2025-07-21T19:31:06Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/hugetlbfs"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk1"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk2"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/cache"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user0"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker/btrfs"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8f116d6e20d2"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/3bd80a3cfd1f"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/etc/libvirt"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/932511d76b14"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/021e49962264"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/0324553c4a51"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/4b5c1f380496"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8a2b3b6d6f06"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/dc93de697727"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/272ed0de122f"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/556ea498fb7b"): no such file or directory
2025-07-21T19:31:15Z D! [inputs.system] Reading users: open /var/run/utmp: no such file or directory
2025-07-21T19:31:16Z D! [outputs.prometheus_client] Wrote batch of 79 metrics in 3.619699ms
2025-07-21T19:31:16Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2025-07-21T19:31:26Z D! [outputs.prometheus_client] Wrote batch of 52 metrics in 5.424221ms
2025-07-21T19:31:26Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/hugetlbfs"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk1"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk2"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/cache"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user0"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker/btrfs"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8f116d6e20d2"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/3bd80a3cfd1f"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/etc/libvirt"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/932511d76b14"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/021e49962264"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/0324553c4a51"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/4b5c1f380496"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8a2b3b6d6f06"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/dc93de697727"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/272ed0de122f"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/556ea498fb7b"): no such file or directory
2025-07-21T19:31:30Z D! [inputs.system] Reading users: open /var/run/utmp: no such file or directory
2025-07-21T19:31:36Z D! [outputs.prometheus_client] Wrote batch of 131 metrics in 8.065137ms
2025-07-21T19:31:36Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/hugetlbfs"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.system] Reading users: open /var/run/utmp: no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk1"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk2"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/cache"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user0"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker/btrfs"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8f116d6e20d2"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/3bd80a3cfd1f"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/etc/libvirt"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/932511d76b14"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/021e49962264"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/0324553c4a51"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/4b5c1f380496"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8a2b3b6d6f06"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/dc93de697727"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/272ed0de122f"): no such file or directory
2025-07-21T19:31:45Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/556ea498fb7b"): no such file or directory
2025-07-21T19:31:46Z D! [outputs.prometheus_client] Wrote batch of 108 metrics in 6.337391ms
2025-07-21T19:31:46Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2025-07-21T19:31:56Z D! [outputs.prometheus_client] Wrote batch of 23 metrics in 2.906695ms
2025-07-21T19:31:56Z D! [outputs.prometheus_client] Buffer fullness: 0 / 10000 metrics
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/hugetlbfs"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk1"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/disk2"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/cache"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user0"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/mnt/user"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/var/lib/docker/btrfs"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8f116d6e20d2"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/3bd80a3cfd1f"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/etc/libvirt"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/932511d76b14"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/021e49962264"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/0324553c4a51"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/4b5c1f380496"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/8a2b3b6d6f06"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/dc93de697727"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/272ed0de122f"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.disk] [SystemPS] => unable to get disk usage ("/run/docker/netns/556ea498fb7b"): no such file or directory
2025-07-21T19:32:00Z D! [inputs.system] Reading users: open /var/run/utmp: no such file or directory
System info
Docker 1.35.2 on UnRAID
Docker
services:
telegraf:
image: telegraf:latest
container_name: telegraf
restart: unless-stopped
privileged: true # Required for SMART access
user: "0:281" # root user, docker group
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys:/rootfs/sys:ro
- /proc:/rootfs/proc:ro
- /etc:/rootfs/etc:ro
- /dev:/dev:ro
- /var/log:/var/log
- /tmp/drive-monitor:/tmp/
- /usr/sbin/smartctl:/usr/sbin/smartctl
- /mnt/user/appdata/telegraf:/etc/telegraf
environment:
- HOST_ETC=/rootfs/etc
- HOST_PROC=/rootfs/proc
- HOST_SYS=/rootfs/sys
ports:
- "9273:9273" # Prometheus metricsSteps to reproduce
- Install telegraf with the given config
- Spin up one of the drives
- Wait for the interval and refresh the /metrics interval
Expected behavior
2 of the drives (the spun down ones) should have the exit_status metric as 2. The other drive should get all the attributes.
Inside the container:
root@c7b3d2c90201:/# smartctl --info --attributes --health -n standby --format=brief /dev/sdc
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.24-Unraid] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
Device is in STANDBY mode, exit(2)
root@c7b3d2c90201:/# smartctl --info --attributes --health -n standby --format=brief /dev/sdb
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.24-Unraid] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: ST18000NT001-3NF101
Serial Number: ZVTFEY8F
LU WWN Device Id: 5 000c50 0e8e32474
Firmware Version: EN01
User Capacity: 18,000,207,937,536 bytes [18.0 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database 7.3/5528
ATA Version is: ACS-4 (minor revision not indicated)
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Jul 21 19:37:53 2025 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Power mode is: ACTIVE or IDLE
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 080 064 044 - 95982944
3 Spin_Up_Time PO---- 090 089 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 113
5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 080 060 045 - 107308876
9 Power_On_Hours -O--CK 094 094 000 - 5471
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 25
18 Unknown_Attribute PO-R-- 100 100 050 - 0
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 098 000 - 60130459662
190 Airflow_Temperature_Cel -O---K 061 045 000 - 39 (Min/Max 34/40)
192 Power-Off_Retract_Count -O--CK 100 100 000 - 5
193 Load_Cycle_Count -O--CK 100 100 000 - 1844
194 Temperature_Celsius -O---K 039 055 000 - 39 (0 17 0 0 0)
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 16
200 Multi_Zone_Error_Rate PO---K 100 100 001 - 0
240 Head_Flying_Hours ------ 100 100 000 - 713 (220 79 0)
241 Total_LBAs_Written ------ 100 253 000 - 66248646320
242 Total_LBAs_Read ------ 100 253 000 - 158263807904
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
/metrics output:
# HELP smart_device_exit_status Telegraf collected metric
# TYPE smart_device_exit_status untyped
smart_device_exit_status{device="sdb",host="big-bertha"} 2
smart_device_exit_status{device="sdc",host="big-bertha"} 2
smart_device_exit_status{device="sde",host="big-bertha"} 2
Actual behavior
The drive sdb should not be exiting with status code 2. It is spinning as shown by running the smartctl command from inside the container.
Additional info
No response