<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>containers on Dhamith&#39;s Notes</title>
    <link>https://blog.dhamith.me/tags/containers/</link>
    <description>Recent content in containers on Dhamith&#39;s Notes</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <copyright>© dhamith93</copyright>
    <lastBuildDate>Tue, 22 Sep 2026 10:23:36 +0530</lastBuildDate><atom:link href="https://blog.dhamith.me/tags/containers/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>systats v0.4.1: Container-Aware Go Module to Scrape Linux Stats</title>
      <link>https://blog.dhamith.me/posts/systats_v0.4.1/</link>
      <pubDate>Tue, 22 Sep 2026 10:23:36 +0530</pubDate>
      
      <guid>https://blog.dhamith.me/posts/systats_v0.4.1/</guid>
      <description>When I first wrote about systats in 2022, it was the metrics code I&amp;rsquo;d pulled out of SyMon: a list of things it could read from /proc. Since then it&amp;rsquo;s grown in a more specific direction: stats for Go services that need to report on themselves and the box they run on, whether that&amp;rsquo;s a health endpoint, a node agent, or an edge device.
This post covers what&amp;rsquo;s changed between v0.</description>
      <content>&lt;p&gt;When I &lt;a href=&#34;https://blog.dhamith.me/posts/systats-go-module-to-collect-linux-system-metrics/&#34;&gt;first wrote about systats&lt;/a&gt;
in 2022, it was the metrics code I&amp;rsquo;d pulled out of &lt;a href=&#34;https://github.com/dhamith93/SyMon&#34;&gt;SyMon&lt;/a&gt;:
a list of things it could read from &lt;code&gt;/proc&lt;/code&gt;. Since then it&amp;rsquo;s grown in a more
specific direction: stats for Go services that need to report on themselves
and the box they run on, whether that&amp;rsquo;s a health endpoint, a node agent, or
an edge device.&lt;/p&gt;
&lt;p&gt;This post covers what&amp;rsquo;s changed between v0.2.0 and v0.4.1. The biggest change
is that it now knows when it&amp;rsquo;s running in a container.&lt;/p&gt;
&lt;h2 id=&#34;container-aware-stats&#34;&gt;Container-aware stats&lt;/h2&gt;
&lt;p&gt;By default, &lt;code&gt;/proc&lt;/code&gt; describes the host, not your container. Without this, a
pod capped at 512MB reports the node&amp;rsquo;s full RAM as its total. Set one field
and &lt;code&gt;GetMemory&lt;/code&gt;, &lt;code&gt;GetCPU&lt;/code&gt; and &lt;code&gt;GetPressure&lt;/code&gt; read the calling process&amp;rsquo;s own
cgroup (v1 and v2, auto-detected) instead:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;syStats&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;:=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;systats&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;New&lt;/span&gt;()
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;syStats&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;ContainerAware&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;mem&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;_&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;:=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;syStats&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;GetMemory&lt;/span&gt;(&lt;span style=&#34;color:#a6e22e&#34;&gt;systats&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Megabyte&lt;/span&gt;)
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;// mem.Limited == true, mem.Total == 512
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;&lt;/span&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;cpu&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;_&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;:=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;syStats&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;GetCPU&lt;/span&gt;()
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;// cpu.AllocatedCores == 0.5 for a Kubernetes &amp;#34;500m&amp;#34; limit
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;// cpu.LoadAvg is a % of that allocation, not of every host core
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It&amp;rsquo;s off by default, and if there&amp;rsquo;s no cgroup limit it falls back to host
numbers. &lt;code&gt;Limited&lt;/code&gt; tells you which you got. It also works for plain systemd
units with &lt;code&gt;MemoryMax=&lt;/code&gt; or &lt;code&gt;CPUQuota=&lt;/code&gt;, not only containers. &lt;code&gt;Load1&lt;/code&gt;/&lt;code&gt;Load5&lt;/code&gt;/&lt;code&gt;Load15&lt;/code&gt;
stay host-wide, since there&amp;rsquo;s no cgroup equivalent.&lt;/p&gt;
&lt;h2 id=&#34;pressure-stall-information&#34;&gt;Pressure stall information&lt;/h2&gt;
&lt;p&gt;CPU percentage and load average can&amp;rsquo;t tell a busy machine from a thrashing
one. PSI can. &lt;code&gt;/proc/pressure&lt;/code&gt; reports how much time tasks spent &lt;em&gt;waiting&lt;/em&gt;
on CPU, memory or I/O:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;p&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;_&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;:=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;syStats&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;GetPressure&lt;/span&gt;()
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;if&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;p&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Available&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;p&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Memory&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Some&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Avg60&lt;/span&gt; &amp;gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;10&lt;/span&gt; {
&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#75715e&#34;&gt;// over 10% of the last minute was spent stalled on memory
&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;&lt;/span&gt;}
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;Some&lt;/code&gt; is the early warning (at least one task stalled); &lt;code&gt;Full&lt;/code&gt; means nothing
ran at all, which is what users feel as slowness. With &lt;code&gt;ContainerAware&lt;/code&gt; on
cgroup v2, you get your own cgroup&amp;rsquo;s pressure (v1 has no PSI, so it falls
back host-wide). On kernels older than 4.20, &lt;code&gt;Available&lt;/code&gt; is false rather
than handing you zeros that look real.&lt;/p&gt;
&lt;h2 id=&#34;no-more-shelling-out-for-stats&#34;&gt;No more shelling out for stats&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;ps&lt;/code&gt;, &lt;code&gt;df&lt;/code&gt;, &lt;code&gt;ip&lt;/code&gt;, &lt;code&gt;lsof&lt;/code&gt; and &lt;code&gt;whereis&lt;/code&gt; are gone, replaced by reads from
&lt;code&gt;/proc&lt;/code&gt; and &lt;code&gt;/sys&lt;/code&gt;, &lt;code&gt;net.Interfaces()&lt;/code&gt; and &lt;code&gt;unix.Statfs&lt;/code&gt;. Stats collection
now works on distroless and scratch images. Two calls still use external
tools: &lt;code&gt;IsServiceRunning&lt;/code&gt; (&lt;code&gt;systemctl&lt;/code&gt;) and the logged-in users list in
&lt;code&gt;GetSystem&lt;/code&gt; (&lt;code&gt;who&lt;/code&gt;). Both now run under &lt;code&gt;LC_ALL=C&lt;/code&gt; with a 5s timeout.&lt;/p&gt;
&lt;p&gt;That second one is what v0.4.1 fixes. On an image without &lt;code&gt;who&lt;/code&gt;, the
&amp;ldquo;executable file not found&amp;rdquo; error message was being parsed as a login
line, producing a user named &lt;code&gt;exec:&lt;/code&gt; logged in from host &lt;code&gt;not&lt;/code&gt;. It now
returns an empty list.&lt;/p&gt;
&lt;h2 id=&#34;new-stats&#34;&gt;New stats&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Disk I/O&lt;/strong&gt;: per-device counters from &lt;code&gt;/proc/diskstats&lt;/code&gt;, with
&lt;code&gt;RatesSince&lt;/code&gt; to turn two samples into throughput, IOPS and iostat-style
&lt;code&gt;%util&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Temperatures&lt;/strong&gt;: per-sensor hwmon readings, useful on Raspberry Pis and
other edge boxes. VMs just return an empty slice.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;TCP connection states&lt;/strong&gt;: counts by state across IPv4 and IPv6, for
spotting &lt;code&gt;TIME_WAIT&lt;/code&gt; buildup.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Protocol counters&lt;/strong&gt;: &lt;code&gt;/proc/net/snmp&lt;/code&gt; and &lt;code&gt;netstat&lt;/code&gt; merged, with an &lt;code&gt;ok&lt;/code&gt;
that separates &amp;ldquo;this kernel doesn&amp;rsquo;t have it&amp;rdquo; from &amp;ldquo;it&amp;rsquo;s zero&amp;rdquo;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Single-process lookup&lt;/strong&gt;: &lt;code&gt;GetProcess(pid)&lt;/code&gt; returns name, state, threads,
open FDs and I/O for one process, e.g. your own via &lt;code&gt;os.Getpid()&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Better CPU topology&lt;/strong&gt;: &lt;code&gt;PhysicalCores&lt;/code&gt; and &lt;code&gt;Sockets&lt;/code&gt;, correct on
multi-socket and hybrid P+E-core machines, plus traditional load averages.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;friendlier-to-real-services&#34;&gt;Friendlier to real services&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Contexts&lt;/strong&gt;: the seven methods that sample, shell out or touch the
network have &lt;code&gt;WithContext&lt;/code&gt; variants. &lt;code&gt;IsServiceRunningWithContext&lt;/code&gt; returns
an error too, so &amp;ldquo;stopped&amp;rdquo; and &amp;ldquo;couldn&amp;rsquo;t check&amp;rdquo; are finally different.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Concurrency&lt;/strong&gt;: a configured &lt;code&gt;SyStats&lt;/code&gt; is safe to share across
goroutines, now documented and covered by a race-detector test.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Injectable paths&lt;/strong&gt;: every path, from &lt;code&gt;ProcPath&lt;/code&gt; to &lt;code&gt;SysClassNetPath&lt;/code&gt;, is
a struct field, so tests can point at fixture trees and still run in
parallel.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tunable CPU sampling&lt;/strong&gt;: &lt;code&gt;CPUSampleWindow&lt;/code&gt; defaults to 300ms; shorten it
if your health check can&amp;rsquo;t wait that long.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Typed parameters&lt;/strong&gt;: &lt;code&gt;Unit&lt;/code&gt;, &lt;code&gt;SortOrder&lt;/code&gt; and &lt;code&gt;CPUMode&lt;/code&gt; are real types, so
a typo like &lt;code&gt;&amp;quot;memroy&amp;quot;&lt;/code&gt; is a compile error instead of a silent fallback.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;see-it&#34;&gt;See it&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;systats&lt;/code&gt; now includes an &lt;code&gt;example/&lt;/code&gt; that renders everything as a single
offline HTML dashboard, with a banner when a cgroup limit is detected.
From a clone of the repo:&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;go run ./example -serve :8080
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;a href=&#34;https://blog.dhamith.me/reports/dashboard-bare-metal.html&#34;&gt;systats example dashboard - bare-metal server&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://blog.dhamith.me/reports/dashboard-vm.html&#34;&gt;systats example dashboard - VM&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;a href=&#34;https://blog.dhamith.me/reports/dashboard-container.html&#34;&gt;systats example dashboard - Docker container&lt;/a&gt;&lt;/p&gt;
&lt;h2 id=&#34;upgrading-from-v020&#34;&gt;Upgrading from v0.2.0&lt;/h2&gt;
&lt;p&gt;There are a few breaking changes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;JSON output uses lowerCamelCase field names (&lt;code&gt;&amp;quot;rxBytes&amp;quot;&lt;/code&gt;, not &lt;code&gt;&amp;quot;RxBytes&amp;quot;&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;code&gt;CPU.NoOfCores&lt;/code&gt; now means logical CPUs; use &lt;code&gt;PhysicalCores&lt;/code&gt; for the old
meaning.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Memory&lt;/code&gt;, &lt;code&gt;Swap&lt;/code&gt; and &lt;code&gt;DiskUsage&lt;/code&gt; sizes are &lt;code&gt;float64&lt;/code&gt;, and &lt;code&gt;Megabyte&lt;/code&gt;/&lt;code&gt;Kilobyte&lt;/code&gt;
values have changed because the old conversions were wrong.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Unit&lt;/code&gt;, &lt;code&gt;SortOrder&lt;/code&gt; and &lt;code&gt;CPUMode&lt;/code&gt; are defined types. Untyped literals still
work, but a &lt;code&gt;string&lt;/code&gt; variable needs &lt;code&gt;systats.Unit(...)&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Disk.Convert&lt;/code&gt; returns an error for unknown units.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;GetTopProcesses&lt;/code&gt; rejects unknown sort orders; use &lt;code&gt;SortByCPU&lt;/code&gt; / &lt;code&gt;SortByMemory&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Go 1.18 or newer is required.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;when-not-to-use-it&#34;&gt;When not to use it&lt;/h2&gt;
&lt;p&gt;systats is deliberately narrow. It&amp;rsquo;s Linux-only and reports on its &lt;em&gt;own&lt;/em&gt;
cgroup. If you need macOS/Windows or want to enumerate other containers,
use &lt;a href=&#34;https://github.com/shirou/gopsutil&#34;&gt;gopsutil&lt;/a&gt;. If you want every raw
&lt;code&gt;/proc&lt;/code&gt; field, use &lt;a href=&#34;https://github.com/prometheus/procfs&#34;&gt;prometheus/procfs&lt;/a&gt;.&lt;/p&gt;
&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;&#34;&gt;&lt;code class=&#34;language-bash&#34; data-lang=&#34;bash&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;go get github.com/dhamith93/systats@v0.4.1
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The full &lt;a href=&#34;https://github.com/dhamith93/systats/blob/main/CHANGELOG.md&#34;&gt;CHANGELOG&lt;/a&gt;
has the details. Issues and PRs welcome: &lt;a href=&#34;https://github.com/dhamith93/systats&#34;&gt;https://github.com/dhamith93/systats&lt;/a&gt;&lt;/p&gt;
</content>
    </item>
    
  </channel>
</rss>
