Setting up a monitoring tool takes an afternoon. The hard part is an alerting setup that warns before a problem grows but doesn’t needlessly wake anyone at 3 a.m. In most environments the tool runs, the mailbox fills with alerts and the real failure gets lost in the noise. Below: which metric really tells you something on Windows and Linux servers, the equivalent counters on the two platforms, and how to derive thresholds from the server’s own behavior instead of fixed numbers.
The purpose of monitoring: failure, performance, capacity
Every metric answers one of three questions. Which one it answers determines the urgency of the alert and who handles it:
- Failure: Is the service working right now? (Service state, port response, disk full, RAID degraded.) Needs immediate action.
- Performance: The service works, but are users experiencing slowness? (Disk latency, CPU queue, memory pressure.) Investigated within hours.
- Capacity: At this trend, when will the resource run out? (Disk growth rate, average CPU increase.) Managed with a weekly or monthly report; it’s a planning matter, not an alert.
A common mistake is to set up a capacity metric like a failure alert. The “disk is 80 percent full” email arrives every morning, after a week nobody opens it, and when the disk really fills up it goes unnoticed. An alert nobody reads isn’t an alert.
Key metrics and what they mean
CPU
High CPU usage on its own isn’t a failure; all cores filling up during a build or backup compression is expected behavior. The real signal is the processor queue: threads ready to run but waiting for the CPU. On Windows, \System\Processor Queue Length consistently exceeding about twice the number of cores, and on Linux, the r column in vmstat output persistently exceeding the number of cores, indicate a CPU bottleneck. On a virtual machine, also look at hypervisor wait: %steal on Linux, CPU Ready on VMware. If these are high, the problem isn’t in the guest operating system but in congestion on the host. Adding vCPUs to the VM can make things worse in this case. The sizing side is in the vCPU, memory and NUMA sizing article.
Memory
The “used memory” number is misleading; both Windows and Linux servers use free memory for the file cache. What to watch is the memory that can be given to applications right away (Available MBytes on Windows, available in the free output on Linux) and whether paging actually goes to disk. On Windows, % Committed Bytes In Use approaching the commit limit shows that you are heading to the point where applications can’t allocate memory even if the page file grows. On Linux, the si/so (swap in/out) columns in vmstat being consistently non-zero is real memory pressure. Part of the swap space sitting used isn’t.
Disk
On the disk side, the metric that tells you the most is latency, i.e. the average duration of each I/O operation. Fill level is a failure and capacity metric; latency is a performance metric. A common starting interpretation: read/write latency under 10 ms is good, 10–20 ms should be watched, and sustained above 20 ms is felt by users. On SSD/NVMe-based storage the expected value is below a millisecond; there, even a persistent latency of a few milliseconds is abnormal. Don’t interpret queue length on its own. Modern storage arrays work efficiently with deep queues; the queue only matters if it rises together with latency.
Network
On the server side, interface utilization, errored and dropped packets and the output queue are monitored. If the dropped packet counter is rising, the cause is usually not bandwidth; it’s the driver, an offload setting or a speed/duplex mismatch on the switch port. Look at the driver version and port settings first. Monitoring the network devices themselves is a separate topic; for choosing a tool, the Zabbix, LibreNMS and PRTG comparison shows the way.
Table of Windows and Linux counter equivalents
| What is measured | Windows (Perfmon) | Linux |
|---|---|---|
| CPU usage | \Processor(_Total)\% Processor Time |
mpstat %usr, %sys |
| CPU queue | \System\Processor Queue Length |
vmstat r |
| I/O wait | Disk latency counters | vmstat wa, iostat %iowait |
| Available memory | \Memory\Available MBytes |
free -m available |
| Memory pressure | \Memory\Pages/sec, % Committed Bytes In Use |
vmstat si/so |
| Disk latency | \LogicalDisk(*)\Avg. Disk sec/Read and /Write |
iostat -x r_await, w_await |
| Disk queue | \LogicalDisk(*)\Current Disk Queue Length |
iostat -x aqu-sz |
| Free space | \LogicalDisk(*)\% Free Space |
df -h, df -i for inodes |
| Network traffic | \Network Interface(*)\Bytes Total/sec |
sar -n DEV rxkB/s, txkB/s |
On Windows, Avg. Disk sec/Read is in seconds; a value of 0.015 means 15 ms. A dashboard that assumes the unit is milliseconds shows latency as “zero.” On Linux, iostat column names vary by sysstat version (await and avgqu-sz in older versions). Inode exhaustion also shows the disk as “full”; if you get write errors while df -h shows free space, the first place to look is df -i. Monitor this separately on servers that produce many small files.
Setting thresholds and the baseline
A static threshold (“alert above 90 percent CPU”) doesn’t mean the same thing on every server. On an application server running batch jobs at night, this value is normal; on a domain controller it is abnormal. The baseline approach first measures the server’s normal, then makes deviation the subject of alerts.
Order of implementation:
- Collect data at one-minute resolution for at least two to four weeks; keep it long enough to cover periodic loads such as month-end or payroll.
- Calculate the 95th percentile for each metric by hour and day. The normal for Monday 09:00 and Sunday 03:00 isn’t the same.
- Put the warning threshold above the baseline, and the critical threshold close to the physical limit of the hardware or service.
- Add a duration condition: not a single measurement but, for example, the threshold being exceeded in “4 of the last 5 measurements.”
On Windows you can start a baseline log independent of the monitoring tool with logman and do spot checks with Get-Counter:
# PowerShell: counter set writing to CSV at 60-second intervals
logman create counter Baseline -si 00:01:00 -f csv -o "C:\PerfLogs\Baseline" -c `
"\Processor(_Total)\% Processor Time" "\System\Processor Queue Length" `
"\Memory\Available MBytes" "\LogicalDisk(*)\Avg. Disk sec/Read" `
"\LogicalDisk(*)\Avg. Disk sec/Write"
logman start Baseline
# Spot check: 12 samples at 5-second intervals
Get-Counter -Counter "\LogicalDisk(*)\Avg. Disk sec/Write" -SampleInterval 5 -MaxSamples 12
On Linux, when the sysstat package is installed and enabled, sar writes historical data to daily files. For spot checks:
vmstat 5 6 # 6 samples at 5-second intervals; the first line is the average since boot
iostat -x 5 3 # latency and queue per device
sar -n DEV 5 3 # traffic per interface
The first report in vmstat and iostat output is the average since boot. For the current state, look at the second and later lines. This is the most common misreading during an incident.
Preventing alert fatigue
Alert fatigue ends with the team no longer reading alerts; the monitoring system runs but is effectively switched off. Rules that prevent it:
- Every alert should require an action: If the person receiving the alert doesn’t know what to do, either the alert is unnecessary or a short response note (runbook) is missing alongside it.
- Dependency definitions: When a branch switch goes down, ten separate alerts shouldn’t be produced for the ten servers behind it; the upstream dependency alert should suppress the downstream ones.
- Hysteresis: The closing threshold should be lower than the opening threshold; otherwise a metric fluctuating around the threshold produces constant open-close cycles.
- Severity and routing: Critical alerts as instant notifications to the on-call person, warnings to a queue during working hours, informational items in reports only.
- Maintenance windows: Alerts should be silenced during planned patching and restarts; for this, the patch calendar should be entered into the monitoring system. For process design, see the server patch management article.
- Regular pruning: Once a month, open the most frequently triggered alerts; question the threshold, the duration condition or whether the alert should exist at all.
Application and service-level checks
The service may not be working while resource metrics are green. CPU, memory and disk are normal; but the IIS application pool has stopped and users see an error page. That’s why a service layer is needed on top of resource monitoring:
- Windows services set to start automatically but not running; the
systemctl --failedoutput on Linux. - Port and protocol checks: not just a TCP connection, but a real transaction such as an HTTP response code, an LDAP query or a SQL session.
- TLS certificate expiry dates; a warning weeks before expiry.
- Event log patterns: on Windows, unexpected shutdowns (6008 and Kernel-Power 41 in the System log) and disk I/O retries (153); on Linux, I/O errors in
journalctl -p erranddmesg. - Hardware: RAID status, power supply and fan alerts via SNMP or Redfish through the vendor management controller (such as iDRAC, iLO, XClarity).
- The results of backup jobs and the age of the last successful backup.
Capacity trend report
The capacity report is a document produced monthly, separate from alerts, that drives decisions for the server and system infrastructure. For each server and volume it includes current usage, the average growth rate over the last 30 and 90 days, and the estimated date the resource will run out at this rate. For disk the calculation is simple:
Example calculation: if a 500 GB data volume has 380 GB used and the average growth over the last 90 days is 1.5 GB per day, the remaining 120 GB runs out in about 80 days. If this is shorter than your purchasing and commissioning time, the expansion must be planned today. The steps for Linux servers are in the disk expansion with LVM article.
Look separately at three trends in the report: the 95th percentile CPU value during working hours rising over months, the floor of available memory dropping, and disk latency worsening without the load increasing. The last one is usually not capacity but a fault on the storage side: a disk starting to fail, a pool close to full, a wrong cache setting. If the load is constant and latency is rising, look at the hardware.
How monitoring metrics are handled together with scheduled maintenance tasks is in the server maintenance checklist article. Designing the monitoring setup and running it continuously falls within the scope of the technical support and scheduled maintenance service. The main reference for Windows counter syntax is the Get-Counter document in Microsoft Learn.
Frequently asked questions
What threshold should be used for CPU usage?
There is no universal value. Evaluate CPU usage together with the processor queue and a duration condition, and set the threshold based on each server’s own baseline data. If high usage doesn’t create a queue, it usually isn’t a problem.
At what value is disk latency considered a problem?
On spinning-disk systems, below 10 ms is considered good, and sustained latency above 20 ms is felt by users. On SSD and NVMe storage the expected value is below a millisecond; even a persistent latency of a few milliseconds should be investigated.
Memory looks almost full on Linux; is there a problem?
Most likely not. Linux uses free memory for the file cache. If the available value in the free output is sufficient and the si/so columns in vmstat aren’t consistently non-zero, there is no memory pressure.
How long should data be collected for a baseline?
At least two to four weeks. Choose a period that covers recurring loads such as month-end close, payroll or periodic reports, and renew the baseline at least once a year.