<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Toml on Tarragon</title><link>https://tarrragon.github.io/blog/tags/toml/</link><description>Recent content in Toml on Tarragon</description><generator>Hugo -- gohugo.io</generator><language>zh-TW</language><copyright>Tarragon (CC BY 4.0)</copyright><lastBuildDate>Mon, 26 Oct 2026 00:00:00 +0800</lastBuildDate><atom:link href="https://tarrragon.github.io/blog/tags/toml/index.xml" rel="self" type="application/rss+xml"/><item><title>多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</title><link>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/</link><pubDate>Mon, 26 Oct 2026 00:00:00 +0800</pubDate><guid>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/</guid><description>&lt;blockquote>
&lt;p>&lt;strong>事件說明&lt;/strong>：2026-10-26，在一台 Apple Silicon 的 macOS 開發機上，用 OrbStack 提供的 Docker Engine 29.8.2（核心 &lt;code>7.2.12-orbstack&lt;/code>，arm64，&lt;a href="https://tarrragon.github.io/blog/backend/knowledge-cards/cgroup/" data-link-title="cgroup（控制群組）" data-link-desc="設定容器的 CPU 與記憶體上限、讀 cAdvisor 的指標、或查容器為什麼被暫停與結束時，查 cgroup 是什麼、v1 與 v2 的差別，以及 Docker 與 Kubernetes 的參數各自對應到它的哪個控制檔">cgroup（控制群組）&lt;/a> v2）建立可拋棄的容器，把多台主機的維護清單從 Terraform 產生 inventory、用 Ansible 部署到兩台主機並執行檢查腳本。受管主機（managed node，Ansible 連過去執行任務的主機）是兩個 Oracle Linux 10.2 容器（systemd 257 當 PID 1，OpenSSH 9.9p1、Python 3.12.15、sudo 1.9.17、dnf-automatic 4.20.0），以 &lt;code>--privileged --cgroupns=private&lt;/code> 執行。控制端（control node，執行 &lt;code>ansible&lt;/code> 與 Terraform 指令的機器）是一個 &lt;code>python:3.12-slim&lt;/code> 容器（Debian 13，OpenSSH 用戶端 10.0p2），裝 ansible-core 2.17.7（Jinja2 3.1.6）與 Terraform 1.9.8，Terraform 的 &lt;code>hashicorp/local&lt;/code> provider（Terraform 操作某一類資源的外掛）是 2.5.2。檢查腳本是 &lt;code>host-check.py&lt;/code> 在 2026-10-26 的版本（作者私有專案的 commit &lt;code>3fc41c0&lt;/code>），原樣複製一份來用，它的原始碼在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制&lt;/a>。&lt;/p>&lt;/blockquote>
&lt;p>這篇把 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍&lt;/a> 的做法在兩台可拋棄的受管主機上從安裝走到清理，一台只當 web、另一台同時是 web 與 db。各節依序是：&lt;/p>
&lt;ul>
&lt;li>受管主機與控制端的容器怎麼建、裝了哪些套件。&lt;/li>
&lt;li>控制端經由 SSH 登入受管主機用的金鑰、&lt;code>known_hosts&lt;/code> 與 sudo。&lt;/li>
&lt;li>Terraform 由主機清單寫出 inventory（Ansible 讀的主機清冊，見 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">Configuration Management（組態管理）&lt;/a>），以及 &lt;code>local_file&lt;/code> 的預設檔案權限。&lt;/li>
&lt;li>變數檔與範本照檢查腳本實際要求的欄位寫，兼任兩種角色的主機怎麼取得兩個角色的入口。&lt;/li>
&lt;li>用 playbook（依序執行多個任務的 YAML 檔）部署檢查腳本與每台的清單：&lt;code>validate&lt;/code> 改用檢查腳本自己的載入函式、在寫出之前就驗出欄位錯誤，以及 &lt;code>--check&lt;/code> 的預覽與實際執行結果不同的那一次。&lt;/li>
&lt;li>以 &lt;code>-b&lt;/code> 與 &lt;code>systemd-run&lt;/code> 對所有主機執行檢查腳本，每台的輸出與 &lt;code>ansible&lt;/code> 指令的 exit code。&lt;/li>
&lt;li>刻意讓一台的門檻變成 &lt;code>WARN&lt;/code>、讓一台的清單格式錯、讓一台連不上時，&lt;code>ansible&lt;/code> 的 exit code 與每台的 &lt;code>rc&lt;/code> 各是什麼。&lt;/li>
&lt;li>容器跟真主機不同的地方，以及在真主機上要另外確認的項目。&lt;/li>
&lt;li>清理。&lt;/li>
&lt;/ul>
&lt;p>值為什麼分成每台相同、依角色不同、每台不同三類，合併為什麼放在部署時，exit code 2 與 4 的意思，這些判斷依據寫在那篇概念文章，這裡只寫套在這次環境上的操作與結果。清單的格式、檢查腳本的白名單與 &lt;code>systemd-run&lt;/code> 各屬性的效果在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本&lt;/a>。這次沒有碰雲端：主機是容器，雲端防火牆那一層的檢查在清單裡寫成要到主控台確認的項目，&lt;code>terraform plan&lt;/code> 對雲端資源的比對也沒有量。&lt;/p></description><content:encoded><![CDATA[<blockquote>
<p><strong>事件說明</strong>：2026-10-26，在一台 Apple Silicon 的 macOS 開發機上，用 OrbStack 提供的 Docker Engine 29.8.2（核心 <code>7.2.12-orbstack</code>，arm64，<a href="/blog/backend/knowledge-cards/cgroup/" data-link-title="cgroup（控制群組）" data-link-desc="設定容器的 CPU 與記憶體上限、讀 cAdvisor 的指標、或查容器為什麼被暫停與結束時，查 cgroup 是什麼、v1 與 v2 的差別，以及 Docker 與 Kubernetes 的參數各自對應到它的哪個控制檔">cgroup（控制群組）</a> v2）建立可拋棄的容器，把多台主機的維護清單從 Terraform 產生 inventory、用 Ansible 部署到兩台主機並執行檢查腳本。受管主機（managed node，Ansible 連過去執行任務的主機）是兩個 Oracle Linux 10.2 容器（systemd 257 當 PID 1，OpenSSH 9.9p1、Python 3.12.15、sudo 1.9.17、dnf-automatic 4.20.0），以 <code>--privileged --cgroupns=private</code> 執行。控制端（control node，執行 <code>ansible</code> 與 Terraform 指令的機器）是一個 <code>python:3.12-slim</code> 容器（Debian 13，OpenSSH 用戶端 10.0p2），裝 ansible-core 2.17.7（Jinja2 3.1.6）與 Terraform 1.9.8，Terraform 的 <code>hashicorp/local</code> provider（Terraform 操作某一類資源的外掛）是 2.5.2。檢查腳本是 <code>host-check.py</code> 在 2026-10-26 的版本（作者私有專案的 commit <code>3fc41c0</code>），原樣複製一份來用，它的原始碼在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</a>。</p></blockquote>
<p>這篇把 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a> 的做法在兩台可拋棄的受管主機上從安裝走到清理，一台只當 web、另一台同時是 web 與 db。各節依序是：</p>
<ul>
<li>受管主機與控制端的容器怎麼建、裝了哪些套件。</li>
<li>控制端經由 SSH 登入受管主機用的金鑰、<code>known_hosts</code> 與 sudo。</li>
<li>Terraform 由主機清單寫出 inventory（Ansible 讀的主機清冊，見 <a href="/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">Configuration Management（組態管理）</a>），以及 <code>local_file</code> 的預設檔案權限。</li>
<li>變數檔與範本照檢查腳本實際要求的欄位寫，兼任兩種角色的主機怎麼取得兩個角色的入口。</li>
<li>用 playbook（依序執行多個任務的 YAML 檔）部署檢查腳本與每台的清單：<code>validate</code> 改用檢查腳本自己的載入函式、在寫出之前就驗出欄位錯誤，以及 <code>--check</code> 的預覽與實際執行結果不同的那一次。</li>
<li>以 <code>-b</code> 與 <code>systemd-run</code> 對所有主機執行檢查腳本，每台的輸出與 <code>ansible</code> 指令的 exit code。</li>
<li>刻意讓一台的門檻變成 <code>WARN</code>、讓一台的清單格式錯、讓一台連不上時，<code>ansible</code> 的 exit code 與每台的 <code>rc</code> 各是什麼。</li>
<li>容器跟真主機不同的地方，以及在真主機上要另外確認的項目。</li>
<li>清理。</li>
</ul>
<p>值為什麼分成每台相同、依角色不同、每台不同三類，合併為什麼放在部署時，exit code 2 與 4 的意思，這些判斷依據寫在那篇概念文章，這裡只寫套在這次環境上的操作與結果。清單的格式、檢查腳本的白名單與 <code>systemd-run</code> 各屬性的效果在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>。這次沒有碰雲端：主機是容器，雲端防火牆那一層的檢查在清單裡寫成要到主控台確認的項目，<code>terraform plan</code> 對雲端資源的比對也沒有量。</p>
<h2 id="前置條件開發機的資源檔案配置與檢查腳本">前置條件：開發機的資源、檔案配置與檢查腳本</h2>
<ul>
<li>
<p><strong>開發機的記憶體</strong>：這次三個容器同時在跑時，<code>docker stats</code> 量到受管主機各用 36 MiB 與 52 MiB、控制端 5 MiB（檢查腳本執行時，systemd 回報那一次執行的記憶體高峰是 70 MiB 到 79 MiB）。這些值依映像檔與正在跑的服務而變；開發機上已經有別的容器在跑時，建容器之前先看 swap 已用多少（macOS 是 <code>sysctl vm.swapusage</code>）與 OrbStack 分到的記憶體上限（在 OrbStack 的設定裡）。</p>
</li>
<li>
<p><strong>主機的數量</strong>：兩台受管主機加一個控制端是這次需要的最小組合：一台只當 web、一台兼任 web 與 db，才看得到依角色取聯集的結果；控制端一個就夠。</p>
</li>
<li>
<p><strong>一個工作目錄</strong>：控制端容器把它掛在 <code>/work</code>，Terraform 的設定、Ansible 的變數檔、範本、產生出來的 inventory 與 Terraform 的 <a href="/blog/devops/infra/knowledge-cards/state/" data-link-title="State（IaC 狀態檔）" data-link-desc="IaC 工具用來記錄每個納管資源在雲端真實樣貌的快照，是比對差異與排定操作順序的依據">State（IaC 狀態檔）</a> 都放在這裡；容器刪掉之後 State 仍在開發機上，清理時才查得到 Terraform 建過什麼。這次的配置：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="ln"> 1</span><span class="cl">.
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">├── host-image/Containerfile       # 受管主機的映像檔
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">├── ctl-image/Containerfile        # 控制端的映像檔
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">└── work/
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    ├── tf/main.tf                 # Terraform：主機清單與寫出 inventory 的 local_file
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    └── ansible/
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">        ├── inventory.yml          # 由 Terraform 寫出，不手改
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">        ├── group_vars/all.yml     # 每台相同的值
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">        ├── group_vars/web.yml     # web 角色的值
</span></span><span class="line"><span class="ln">10</span><span class="cl">        ├── group_vars/db.yml      # db 角色的值
</span></span><span class="line"><span class="ln">11</span><span class="cl">        ├── host_vars/mh-web1.yml  # 只有 mh-web1 不同的值
</span></span><span class="line"><span class="ln">12</span><span class="cl">        ├── host.toml.j2           # 清單的範本
</span></span><span class="line"><span class="ln">13</span><span class="cl">        ├── files/host-check.py    # 檢查腳本（由原始碼那一篇的區塊接成）
</span></span><span class="line"><span class="ln">14</span><span class="cl">        ├── deploy.yml             # 部署腳本與清單的 playbook
</span></span><span class="line"><span class="ln">15</span><span class="cl">        └── check.sh               # 對每台執行檢查的 ad-hoc 指令（不寫 playbook、直接執行單一模組）</span></span></code></pre></div><p>下面各節程式碼區塊第一行的註解寫著檔名，照那個路徑在開發機上建檔，路徑的起點依檔案分三種：</p>
<ul>
<li><code>host-image/…</code>、<code>ctl-image/…</code>：從 <code>work/</code> 的上一層起算。兩個映像檔的建置目錄不必掛進控制端，所以不放進 <code>work/</code>。</li>
<li><code>tf/main.tf</code>：從工作目錄 <code>work/</code> 起算。</li>
<li>其餘的（<code>group_vars/…</code>、<code>host.toml.j2</code>、<code>deploy.yml</code>、<code>check.sh</code>）：從 <code>work/ansible/</code> 起算。</li>
</ul>
<p><code>ansible/</code> 與 <code>tf/</code> 目錄要在起控制端容器之前建好。<code>inventory.yml</code> 跟 <code>group_vars/</code>、<code>host_vars/</code> 放在同一個目錄，是 Ansible 載入這兩個目錄的條件：Ansible 從 inventory 檔所在的目錄（以及 playbook 所在的目錄）找 <code>group_vars/</code> 與 <code>host_vars/</code>。</p>
</li>
<li>
<p><strong>檢查腳本</strong>：<code>host-check.py</code> 的原始碼在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</a>，把那一篇的 <code>python</code> 區塊照它開頭寫的接法接起來，存成 <code>files/host-check.py</code>。換成自己的檢查腳本時，這篇的步驟用到的腳本行為有下面這幾項，替代的腳本要全部滿足，或照它的行為改寫對應的步驟：</p>
<ul>
<li>有一個只載入與驗證清單、不執行檢查的函式或模式，清單格式錯時以非零結束（<code>host-check.py</code> 是 <code>load_manifest()</code>，格式錯時丟出例外）；部署時的 <code>validate</code> 用它。</li>
<li>exit code 0 代表沒有 <code>WARN</code>、1 代表至少一項 <code>WARN</code>、2 代表清單格式錯；跨主機的輸出靠每台的 <code>rc</code> 分出這三種情形（<code>rc</code> 是 Ansible 回報的、那台主機上指令的 return code，執行檢查時就是檢查腳本的 exit code）。</li>
<li>結尾一定印一行 <code>summary:</code>；腳本沒有開始或沒有跑完時，靠這一行分辨。</li>
<li>接受 <code>--manifest &lt;路徑&gt;</code> 指定清單的位置。</li>
<li><code>--no-state</code> 不讀也不寫結果檔。</li>
<li>結果檔放在 <code>/var/lib/host-check/</code>，由 <code>systemd-run</code> 的 <code>StateDirectory=host-check</code> 建立。</li>
<li>在沙箱底下跑得完：<code>ProtectSystem=strict</code>、只有 <code>ReadWritePaths</code> 列的 <code>/var/log</code> 與 <code>/var/cache/dnf</code> 兩個目錄（加上 <code>StateDirectory</code>）可寫、<code>IPAddressDeny=any</code> 擋掉所有網路連線。</li>
<li>給 <code>runpy</code> 載入時：模組層級沒有副作用、<code>__name__</code> 不是 <code>__main__</code> 時不執行主程式、載入函式以丟出例外表達格式錯；主機的 Python 是 3.11 以上（<code>tomllib</code>）。</li>
<li>清單的欄位：〈變數檔與範本：檢查腳本實際要求的欄位、兼任兩種角色的主機〉那一節列的欄位與允許值是 <code>host-check.py</code> 自己的規格，換腳本時範本換成那支腳本要求的欄位。</li>
</ul>
<p>換成自己的腳本時，正文的預期輸出（<code>PASS</code>、<code>WARN</code>、<code>MANUAL</code> 的行格式、<code>since:</code> 與 <code>DELTA</code>、<code>manifest error:</code> 的訊息、<code>[vars]</code> 的替換）以那支腳本自己的輸出為準。</p>
</li>
<li>
<p><strong>容器、network 與映像檔的名稱一律 <code>mh-</code> 開頭</strong>：清理時 <code>docker ps -a --filter name=mh-</code> 這一類查詢列得出這次建的全部資源，開發機上其他專案的容器不會被誤刪。</p>
</li>
<li>
<p><strong>把這個 lab 固定成自動化測試時</strong>：可用的工具是 molecule 的 docker driver，它拿容器當受管主機，依序建立容器、套用 playbook（converge）、執行驗證。這次沒有用 molecule，每一步都手動執行。</p>
</li>
</ul>
<h2 id="受管主機與控制端的容器oracle-linux-10-加-systemd-與-sshdpython312-slim-加-ansible-core-與-terraform">受管主機與控制端的容器：Oracle Linux 10 加 systemd 與 sshd、python:3.12-slim 加 ansible-core 與 Terraform</h2>
<p>受管主機的映像檔以 <code>oraclelinux:10</code> 為底，裝 sshd、sudo、Python 與檢查腳本會查的 dnf-automatic，建一個 <code>ops</code> 帳號當 Ansible 登入的帳號：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c"># host-image/Containerfile：受管主機</span><span class="err">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="err"></span><span class="k">FROM</span><span class="s"> oraclelinux:10</span><span class="err">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="err"></span><span class="k">RUN</span> dnf -y install systemd procps-ng iproute openssh-server sudo python3 dnf-automatic dnf-plugins-core <span class="se">\
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> dnf clean all <span class="se">\
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> systemctl mask systemd-logind.service getty.target console-getty.service <span class="se">\
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> systemctl <span class="nb">enable</span> sshd.service <span class="se">\
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> useradd -m ops <span class="se">\
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> <span class="nb">echo</span> <span class="s1">&#39;ops ALL=(ALL) NOPASSWD: ALL&#39;</span> &gt; /etc/sudoers.d/ops <span class="o">&amp;&amp;</span> chmod <span class="m">0440</span> /etc/sudoers.d/ops<span class="err">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="err"></span><span class="k">STOPSIGNAL</span><span class="s"> SIGRTMIN+3</span><span class="err">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="err"></span><span class="k">CMD</span> <span class="p">[</span><span class="s2">&#34;/usr/sbin/init&#34;</span><span class="p">]</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c"># ctl-image/Containerfile：控制端</span><span class="err">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="err"></span><span class="k">FROM</span><span class="s"> python:3.12-slim</span><span class="err">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="err"></span><span class="k">ARG</span> <span class="nv">TF_VERSION</span><span class="o">=</span><span class="m">1</span>.9.8<span class="err">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="err"></span><span class="c"># TARGETARCH 由 docker build 依開發機的架構填入（這台是 arm64），Terraform 的壓縮檔名照它選</span><span class="err">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="err"></span><span class="k">ARG</span> TARGETARCH<span class="err">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="err"></span><span class="k">RUN</span> apt-get update <span class="o">&amp;&amp;</span> apt-get install -y --no-install-recommends openssh-client curl unzip <span class="se">\
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> rm -rf /var/lib/apt/lists/* <span class="se">\
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> pip install --no-cache-dir ansible-core<span class="o">==</span>2.17.7 <span class="se">\
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> curl -fsSLo /tmp/tf.zip https://releases.hashicorp.com/terraform/<span class="si">${</span><span class="nv">TF_VERSION</span><span class="si">}</span>/terraform_<span class="si">${</span><span class="nv">TF_VERSION</span><span class="si">}</span>_linux_<span class="si">${</span><span class="nv">TARGETARCH</span><span class="si">}</span>.zip <span class="se">\
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="se"></span>    <span class="o">&amp;&amp;</span> unzip /tmp/tf.zip terraform -d /usr/local/bin <span class="o">&amp;&amp;</span> rm /tmp/tf.zip<span class="err">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="err"></span><span class="k">WORKDIR</span><span class="s"> /work</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 開發機上，在 work/ 的上一層執行；-f 指定檔名，因為檔名是 Containerfile 而 docker build 預設找 Dockerfile</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker build -f host-image/Containerfile -t mh-host:ol10 host-image
</span></span><span class="line"><span class="ln">3</span><span class="cl">docker build -f ctl-image/Containerfile -t mh-ctl:1 ctl-image
</span></span><span class="line"><span class="ln">4</span><span class="cl">docker run --rm mh-ctl:1 sh -c <span class="s1">&#39;ansible --version | head -1; terraform version | head -1&#39;</span></span></span></code></pre></div><p>最後一條印出 <code>ansible [core 2.17.7]</code> 與 <code>Terraform v1.9.8</code>。這台開發機上 <code>oraclelinux:10</code> 已經在本機，受管主機的映像檔建了 23 秒；控制端要下載 pip 套件與 Terraform 的壓縮檔，建了 101 秒。兩個時間依網路與本機的快取而變。</p>
<p>受管主機的 Containerfile 最後兩行讓 systemd 當容器的 PID 1：<code>CMD [&quot;/usr/sbin/init&quot;]</code> 啟動 systemd，<code>STOPSIGNAL SIGRTMIN+3</code> 讓 <code>docker stop</code> 送出 systemd 認得的關機訊號。systemd 收到 SIGRTMIN+3 才照正常的關機流程停下各個 unit；Docker 預設送的是 SIGTERM，systemd 不把它當成關機，<code>docker stop</code> 會等到逾時再強制結束容器（依 systemd 文件對訊號的說明推論，沒有實跑拿掉這一行的結果）。</p>
<p><strong>這次遇到的錯誤</strong>：第一次建受管主機的映像檔時套件名寫成 <code>'dnf-command(needs-restarting)'</code>，dnf 回 <code>Unable to find a match</code>，在 Oracle Linux 10 上，檢查腳本用的 <code>dnf needs-restarting</code> 子命令由 <code>dnf-plugins-core</code> 提供（外掛檔案在它依賴的 <code>python3-dnf-plugins-core</code> 裡），<code>oraclelinux:10</code> 的映像檔已經裝了它；另一個獨立的 <code>/usr/bin/needs-restarting</code> 指令屬於 <code>yum-utils</code>（2026-10-27 在 <code>oraclelinux:10</code> 容器裡用 <code>rpm -qf</code> 查到，Oracle Linux 10.2）。Containerfile 仍把 <code>dnf-plugins-core</code> 寫進安裝清單，讓檢查腳本依賴的套件明示在這一行：換成沒有預裝它的底層映像檔時，建映像檔那一步會裝上它，不會漏掉。</p>
<p>換成 VM 或別的發行版、不開特權、sudo 要密碼、改用別的安裝方式時的結果，在〈這一節的其他做法：容器或 VM、作業系統、遮蔽的 unit、privileged、sudo 密碼、安裝版本與雜湊、金鑰密碼〉。</p>
<p>兩台受管主機與控制端接在同一個 Docker network 上，控制端用容器名就連得到受管主機：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl">docker network create mh-net
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker run -d --name mh-web1  --hostname mh-web1  --network mh-net --privileged --cgroupns<span class="o">=</span>private mh-host:ol10
</span></span><span class="line"><span class="ln">3</span><span class="cl">docker run -d --name mh-both1 --hostname mh-both1 --network mh-net --privileged --cgroupns<span class="o">=</span>private mh-host:ol10
</span></span><span class="line"><span class="ln">4</span><span class="cl">docker <span class="nb">exec</span> mh-web1 systemctl is-system-running --wait    <span class="c1"># 等 systemd 啟動完成，印 running</span>
</span></span><span class="line"><span class="ln">5</span><span class="cl">docker <span class="nb">exec</span> mh-both1 systemctl is-system-running --wait</span></span></code></pre></div><p><code>docker network create</code> 另建一個 network，因為 Docker 預設的 bridge network 不提供容器名的 DNS 解析，接在上面的容器只能用 IP 互連；自建的 bridge network 由 Docker 內建的 DNS 把容器名解析成位址（依 Docker 文件〈Bridge network driver〉，沒有在預設的 bridge 上實跑）。</p>
<p><code>--privileged --cgroupns=private</code> 讓容器裡的 systemd 管得到自己的 cgroup，<code>systemd-run</code> 才建得出暫時的 unit（transient unit：執行時由 <code>systemd-run</code> 建立、不寫成 unit 檔的 unit）。Docker 在 cgroup v2 的主機上，cgroup namespace 的預設就是 <code>private</code>，所以 systemd 能否當 PID 1 啟動，取決於 <code>--privileged</code>，<code>--cgroupns=private</code> 只是把預設寫明（依 <code>docker run</code> 參考文件推論）。拿掉 <code>--privileged</code> 時，容器裡的 systemd 管不到自己的 cgroup，PID 1 起不來，或 <code>systemd-run</code> 建不出暫時的 unit（從 systemd 文件〈The Container Interface〉對 cgroup 階層的要求推論，沒有實跑）。<a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a> 在同一種容器裡量沙箱的效果時用的就是這組參數。<code>--privileged</code> 的代價與不開特權的做法，在〈這一節的其他做法：容器或 VM、作業系統、遮蔽的 unit、privileged、sudo 密碼、安裝版本與雜湊、金鑰密碼〉。</p>
<p><code>systemctl is-system-running --wait</code> 等到 systemd 的啟動結束才印出狀態：<code>running</code> 代表啟動完成、沒有失敗的 unit，<code>degraded</code> 代表有 unit 失敗，用 <code>systemctl --failed</code> 查是哪一個。不加 <code>--wait</code>、緊接在 <code>docker run</code> 之後執行時，這次印的是 <code>initializing</code>、exit code 1，systemd 還在啟動，sshd 可能還沒開始接聽。</p>
<p>兩台各再做三件事，這三件在真主機上分別由建立主機時的設定與服務自己的部署負責，這裡用 <code>docker exec</code> 代替：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># 開發機上執行；&lt;暫存目錄&gt; 換成工作目錄以外的一個暫存目錄，金鑰的用法在〈控制端連到受管主機的 SSH：金鑰、known_hosts 與 sudo〉</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">mkdir -p &lt;暫存目錄&gt;/keys    <span class="c1"># ssh-keygen 不建上層目錄</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">ssh-keygen -q -t ed25519 -N <span class="s1">&#39;&#39;</span> -C mh-walkthrough -f &lt;暫存目錄&gt;/keys/id_ed25519
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="k">for</span> h in mh-web1 mh-both1<span class="p">;</span> <span class="k">do</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">  docker <span class="nb">exec</span> <span class="nv">$h</span> install -d -m <span class="m">0700</span> -o ops -g ops /home/ops/.ssh
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">  docker cp &lt;暫存目錄&gt;/keys/id_ed25519.pub <span class="nv">$h</span>:/home/ops/.ssh/authorized_keys
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">  docker <span class="nb">exec</span> <span class="nv">$h</span> sh -c <span class="s1">&#39;chown ops:ops /home/ops/.ssh/authorized_keys &amp;&amp; chmod 0600 /home/ops/.ssh/authorized_keys&#39;</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">  docker <span class="nb">exec</span> <span class="nv">$h</span> systemctl <span class="nb">enable</span> --now dnf-automatic.timer
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">  docker <span class="nb">exec</span> <span class="nv">$h</span> systemd-run --unit<span class="o">=</span>stand-in-http python3 -m http.server <span class="m">80</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="k">done</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">docker <span class="nb">exec</span> mh-both1 systemd-run --unit<span class="o">=</span>stand-in-postgres python3 -m http.server <span class="m">5432</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">docker <span class="nb">exec</span> mh-both1 ss -H -l -n -t</span></span></code></pre></div><ul>
<li><strong>公鑰</strong>：<code>ssh-keygen</code> 在暫存目錄產生一把新的金鑰。暫存目錄要在工作目錄以外：工作目錄整個掛進控制端容器，私鑰放在裡面的話會跟著掛進去，而控制端只需要唯讀掛載的那一份金鑰目錄。<code>keys</code> 目錄不存在時它以 <code>Saving key … failed: No such file or directory</code> 失敗，所以先 <code>mkdir -p</code>。<code>-N ''</code> 讓私鑰不設密碼，控制端執行 Ansible 時才不必輸入。雲端主機在建立時由 cloud-init 把公鑰放進預設帳號，這裡用 <code>docker cp</code> 手動放。</li>
<li><strong>dnf-automatic 的 timer</strong>：清單要求它 <code>enabled</code> 而且 <code>active</code>，真主機上由部署腳本啟用。</li>
<li><strong>代替服務的接聽程式</strong>：web 角色的入口是 80，db 角色是 5432，檢查腳本確認那個連接埠有程序在聽。這次沒有裝 nginx 與 PostgreSQL，用 <code>python3 -m http.server</code> 在兩個連接埠上接聽；<code>systemd-run --unit=</code> 建的是暫時的 unit，容器重啟之後就不在，〈一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc〉那一節正好用到這個性質。</li>
</ul>
<p>最後一條在 <code>mh-both1</code> 上列出 <code>0.0.0.0:22</code>、<code>[::]:22</code>、<code>0.0.0.0:80</code> 與 <code>0.0.0.0:5432</code>，另有 Docker 內建 DNS 在 <code>127.0.0.11</code> 上的一個 TCP 連接埠（它的另一個連接埠是 UDP，不在 <code>-t</code> 的輸出裡）；Docker 內建 DNS 的位址在 loopback 上，檢查腳本的 <code>listen_audit</code> 只比對 loopback 以外的位址，所以 DNS 的連接埠不算多開的連接埠。</p>
<h3 id="這一節的其他做法容器或-vm作業系統遮蔽的-unitprivilegedsudo-密碼安裝版本與雜湊金鑰密碼">這一節的其他做法：容器或 VM、作業系統、遮蔽的 unit、privileged、sudo 密碼、安裝版本與雜湊、金鑰密碼</h3>
<ul>
<li><strong>受管主機用容器</strong>：容器建得快、刪得乾淨，代價是沙箱、SELinux、logind 這幾項跟真主機不同（〈容器跟真主機不同的地方，以及在真主機上要另外確認的項目〉）。換成完整的 VM（Lima、UTM、multipass）或雲端主機時，那一節列的 SELinux、logind 的 session 與持久的 journal 這幾項就量得到，代價是建立與清理都比較慢、雲端主機要計費（推論，這次沒有用 VM 跑）。</li>
<li><strong>受管主機的作業系統</strong>：檢查腳本的 <code>[updates]</code> 段讀的是 dnf-automatic 的設定檔（INI 格式的 <code>[commands]</code> 段），<code>reboot_check</code> 用 <code>dnf needs-restarting -r</code>，所以選 Oracle Linux；同為 dnf 系的 Rocky Linux、AlmaLinux 同樣適用（推論，沒有實跑）。換成 <code>debian:12</code> 這類 Debian 系的映像檔可以少下載一些，代價是 Debian 上沒有 dnf-automatic 的設定檔、timer 與 <code>dnf</code> 指令，<code>[updates]</code> 那幾項（timer、config、reboot）都不會是 <code>PASS</code>，<code>[updates]</code> 段要照 unattended-upgrades 的設定改寫，檢查腳本也要跟著改（從腳本的程式碼推的，沒有實跑）。</li>
<li><strong>systemd 當 PID 1 與遮蔽的 unit</strong>：檢查要用 <code>systemd-run</code> 套沙箱，<code>[updates]</code> 也要查 <a href="/blog/linux/dotfile/knowledge-cards/systemd-timer/" data-link-title="systemd timer" data-link-desc="排程的工作沒有在預期的時間執行、主機關機期間錯過的排程不知道會不會補跑、要在 cron 與 systemd timer 之間選一種寫定期工作，或讀到 *.timer 檔與 systemctl list-timers 的輸出時">systemd timer</a> 的狀態，所以容器裡要有 systemd。<code>systemctl mask</code> 那幾個 unit 是容器裡用不到的登入與終端機服務；遮蔽 <code>systemd-logind</code> 的副作用寫在〈容器跟真主機不同的地方，以及在真主機上要另外確認的項目〉。不遮蔽時，logind 會替每次 SSH 登入建一個 session scope，Ansible 模組的紀錄落在那個 scope 底下，<code>log/sshd</code> 的最新一筆就不再是 Ansible 的紀錄（推論，這次沒有實跑不遮蔽的版本）。</li>
<li><strong><code>--privileged</code> 的代價與不開特權的做法</strong>：<code>--privileged</code> 的代價在 Linux 開發機上最明顯：這種容器拿得到主機上所有的裝置與 Linux capability，約等於主機上的 root（<a href="/blog/backend/knowledge-cards/privileged-container/" data-link-title="Privileged Container（特權容器）" data-link-desc="在 Pod 設定裡看到 securityContext.privileged=true、要判斷某個除錯或系統元件為什麼需要它，或評估誰能建這種 Pod 等於持有節點 root 權限時，查特權容器能做什麼與它的權限邊界">Privileged Container（特權容器）</a>；Docker 的 <code>--privileged</code> 跟卡上寫的 <code>securityContext.privileged</code> 是同一種權限）。這次的 Docker Engine 跑在 OrbStack 的 Linux VM 裡，拿到的是那台 VM 的權限，不是 macOS 本身。不想開特權時，Podman 的 <code>--systemd=always</code> 不需要 privileged 也能讓 systemd 當 PID 1（依 Podman 文件，沒有實跑）。</li>
<li><strong><code>ops</code> 帳號的 sudo 不要密碼</strong>：Ansible 的 become（<code>-b</code>，讓 Ansible 登入之後經由 sudo 切成 root）因此不必另外給密碼。sudo 要密碼的主機，<code>ansible</code> 與 <code>ansible-playbook</code> 要加 <code>-K</code>（<code>--ask-become-pass</code>）在執行時輸入，或把密碼放進 Ansible Vault（Ansible 內建的加密功能，把變數檔加密存放、執行時用密碼解開）加密的變數；不給的話 become 那一步失敗（依 Ansible 文件〈Understanding privilege escalation: become〉，沒有實跑）。由排程執行時沒有人輸入，<code>-K</code> 用不了；改用 Vault 時，解開 Vault 的密碼檔要放在控制端上，控制端被入侵時密碼檔與私鑰一起被拿走，風險的範圍沒有變小（推論）。</li>
<li><strong>ansible-core 與 Terraform 固定版本</strong>：ansible-core 用 <code>pip install ansible-core==2.17.7</code>，Terraform 下載 1.9.8 的壓縮檔。改用 apt、pipx 或 HashiCorp 的套件庫安裝時拿到的版本可能不同，版本不同時 <code>ansible</code> 與 <code>terraform</code> 印出的格式與本文的輸出不保證相同（從版本差異推論，沒有實跑其他版本）。</li>
<li><strong>Terraform 的壓縮檔沒有驗證雜湊</strong>：HashiCorp 在同一個目錄放了 <code>terraform_&lt;版本&gt;_SHA256SUMS</code> 與它的簽章，下載之後用 <code>sha256sum -c</code> 比對，可以確認下載到的檔案沒有被換掉；這次沒有做這一步（依 HashiCorp 的下載頁，沒有實跑）。</li>
<li><strong>金鑰不設密碼</strong>：替代做法是設密碼，在控制端起 <code>ssh-agent</code> 並 <code>ssh-add</code> 解鎖一次：私鑰檔外洩時多一層密碼保護，代價是每次開一個新的 shell 都要先解鎖（推論，沒有實跑）。</li>
</ul>
<h2 id="控制端連到受管主機的-ssh金鑰known_hosts-與-sudo">控制端連到受管主機的 SSH：金鑰、known_hosts 與 sudo</h2>
<p>控制端用的金鑰是〈受管主機與控制端的容器：Oracle Linux 10 加 systemd 與 sshd、python:3.12-slim 加 ansible-core 與 Terraform〉那一節在暫存目錄產生的那一把，開發機上自己的金鑰沒有拿來用：這把金鑰只授權給這兩個可拋棄的容器，清理時連同容器一起刪掉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 開發機上；&lt;工作目錄&gt; 是〈前置條件：開發機的資源、檔案配置與檢查腳本〉的工作目錄，&lt;暫存目錄&gt; 是產生金鑰時用的那一個</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker run -d --name mh-ctl --network mh-net -v &lt;工作目錄&gt;:/work -v &lt;暫存目錄&gt;/keys:/keys:ro mh-ctl:1 sleep infinity
</span></span><span class="line"><span class="ln">3</span><span class="cl">docker <span class="nb">exec</span> mh-ctl sh -c <span class="s1">&#39;mkdir -p ~/.ssh &amp;&amp; cp /keys/id_ed25519 ~/.ssh/ &amp;&amp; chmod 600 ~/.ssh/id_ed25519
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="s1">  for h in mh-web1 mh-both1; do ssh-keyscan -t ed25519 $h; done &gt; ~/.ssh/known_hosts
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="s1">  for h in mh-web1 mh-both1; do ssh ops@$h &#34;id; sudo -n id -u&#34;; done&#39;</span></span></span></code></pre></div><p><code>-v &lt;工作目錄&gt;:/work</code> 把開發機上的工作目錄掛進控制端；金鑰的目錄唯讀掛載，再複製一份進容器改權限，因為 ssh 拒絕權限比 0600 寬的私鑰，而掛載進來的檔案權限照開發機那一份。私鑰複製到 <code>~/.ssh/id_ed25519</code>，是 ssh 沒有指定金鑰時預設會試的路徑之一，Ansible 經由 ssh 連線時也就用這一把，inventory 不必另外寫 <code>ansible_ssh_private_key_file</code>。最後那個迴圈在 <code>mh-web1</code> 上印出 <code>uid=1000(ops)</code> 與 <code>0</code>：<code>ops</code> 登入成功，<code>sudo -n</code>（不問密碼，要密碼就直接失敗）切成了 root。這次實跑時這一條只連了 <code>mh-web1</code>，<code>mh-both1</code> 的登入與 sudo 由後面的 <code>ansible -b</code> 指令確認。</p>
<p><code>ssh-keyscan</code> 把兩台主機出示的 host key 直接寫進 <code>known_hosts</code>，等於第一次連線時不比對指紋就信任（<a href="/blog/devops/infra/knowledge-cards/ssh-host-key/" data-link-title="SSH Host Key 與 known_hosts（主機金鑰）" data-link-desc="ssh 第一次連線問要不要信任指紋、出現 REMOTE HOST IDENTIFICATION HAS CHANGED 警告，或主機重建之後連不上時，查 host key 怎麼證明主機身分、known_hosts 記了什麼，以及首次信任的風險">SSH Host Key 與 known_hosts（主機金鑰）</a> 卡寫了首次信任的風險）。在一個只有這三個容器的 Docker network 裡，連線的另一端只可能是這次建的容器，所以這次接受這個風險。改用自己的金鑰、或改用別的方式寫入 <code>known_hosts</code> 時的結果，在〈這一節的其他做法：自己的金鑰與寫入 known_hosts 的方式〉。</p>
<p>之後的 Terraform 與 Ansible 指令都在控制端容器裡的 shell 執行，從開發機進入那個 shell：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 開發機上；-w 把 shell 的工作目錄設在 /work/ansible</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker <span class="nb">exec</span> -it -w /work/ansible mh-ctl bash</span></span></code></pre></div><p>下文寫「控制端容器裡」的指令都在這個 shell 執行，工作目錄是 <code>/work/ansible</code>；Terraform 那幾條在 <code>/work/tf</code>，區塊裡先 <code>cd /work/tf</code>。</p>
<h3 id="這一節的其他做法自己的金鑰與寫入-known_hosts-的方式">這一節的其他做法：自己的金鑰與寫入 known_hosts 的方式</h3>
<ul>
<li><strong>用自己的金鑰</strong>：用自己的金鑰也連得上，代價是要把自己的私鑰掛進控制端容器，控制端容器裡的任何程式都讀得到它。</li>
<li><strong><code>ssh-keyscan</code> 寫進 <code>known_hosts</code></strong>（這次的做法）：之後的連線都會比對 host key，主機重建、host key 換了時 ssh 拒絕連線。</li>
<li><strong>不寫 <code>known_hosts</code></strong>：Ansible 預設檢查 host key，第一次連到沒有紀錄的主機時沒有人回答確認的提問，那一台失敗；在 <code>ansible.cfg</code> 設 <code>host_key_checking = False</code> 可以跳過，代價是之後每次連線都不比對，連到被冒充的主機也不會發現（依 Ansible 文件〈Connection methods and details〉的〈Managing host key checking〉，沒有實跑）。</li>
<li><strong><code>StrictHostKeyChecking=accept-new</code></strong>（OpenSSH 7.6 起）：第一次連到沒有紀錄的主機時自動把 host key 記進 <code>known_hosts</code>，之後 host key 變了照樣拒絕連線；首次連線同樣不比對指紋，跟 <code>ssh-keyscan</code> 承擔的是同一個風險，差別是不必先跑一次掃描（依 ssh_config 的文件，沒有實跑）。</li>
<li><strong>比對主控台或序列主控台印出的指紋再寫入</strong>：真主機建立時，業者的主控台或序列主控台的開機輸出印得出 host key 的指紋，比對過 <code>ssh-keyscan</code> 取回的 host key 再寫進 <code>known_hosts</code>，第一次連線也不必無條件信任。代價是每建一台主機都要人工比對一次（這次沒有真主機，沒有實跑）。</li>
<li><strong>SSH 憑證：只寫 CA 公鑰</strong>：用 <a href="/blog/devops/infra/knowledge-cards/ssh-certificate/" data-link-title="SSH Certificate（SSH 憑證）" data-link-desc="主機多、人員常異動而逐台收回 authorized_keys 跟不上，或要讓 SSH 登入權限自動到期時，查 SSH CA 簽發短期憑證的運作方式與導入時要決定的事">SSH Certificate（SSH 憑證）</a> 簽主機金鑰的環境，<code>known_hosts</code> 只要寫一行 CA 的公鑰，新建的主機不必逐台加紀錄；代價是要先建好簽發主機憑證的 CA 與流程（從機制推論，沒有實跑）。</li>
</ul>
<h2 id="由-terraform-產生-inventoryinitplanapply-與-local_file-的檔案權限">由 Terraform 產生 inventory：init、plan、apply 與 local_file 的檔案權限</h2>
<p>Terraform 的設定（<a href="/blog/devops/infra/knowledge-cards/hcl/" data-link-title="HCL" data-link-desc="Terraform 使用的宣告式設定語言，用 resource block 描述基礎設施的目標狀態，工具負責收斂差異">HCL</a> 寫成）照 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈主機由 Terraform 建立時由它的輸出產生 inventory〉的寫法，主機改成這次的兩個容器，<code>all</code> 群組（Ansible 內建、包含 inventory 裡每一台主機的群組）加上 Ansible 登入用的帳號。跟概念文章那份設定不同的兩處：每台主機的鍵名 <code>public_ip</code> 改成 <code>address</code>，因為這次填的是容器名、不是 IP；<code>filename</code> 指向 <code>ansible/</code> 目錄，inventory 寫在 Ansible 的變數檔旁邊。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># tf/main.tf
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"></span><span class="k">terraform</span> {
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">  <span class="k">required_providers</span> {
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="n">    local</span> <span class="o">=</span><span class="n"> { source</span> <span class="o">=</span><span class="n"> &#34;hashicorp/local&#34;, version</span> <span class="o">=</span> <span class="s2">&#34;2.5.2&#34;</span> }
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">  }
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">}<span class="c1">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="c1">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="c1"># 實際的設定裡，address 來自主機資源的屬性（例如 public_ip）；這裡用容器名代替
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="c1"></span><span class="k">locals</span> {
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="n">  hosts</span> <span class="o">=</span> {
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="n">    &#34;mh-web1&#34;</span>  <span class="o">=</span><span class="n"> { roles</span> <span class="o">=</span><span class="n"> [&#34;web&#34;], address</span> <span class="o">=</span> <span class="s2">&#34;mh-web1&#34;</span> }
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="n">    &#34;mh-both1&#34;</span> <span class="o">=</span><span class="n"> { roles</span> <span class="o">=</span><span class="n"> [&#34;web&#34;, &#34;db&#34;], address</span> <span class="o">=</span> <span class="s2">&#34;mh-both1&#34;</span> }
</span></span><span class="line"><span class="ln">13</span><span class="cl">  }
</span></span><span class="line"><span class="ln">14</span><span class="cl">}
</span></span><span class="line"><span class="ln">15</span><span class="cl">
</span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="k">resource</span> <span class="s2">&#34;local_file&#34; &#34;inventory&#34;</span> {
</span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="n">  filename</span>        <span class="o">=</span> <span class="s2">&#34;${path.module}/../ansible/inventory.yml&#34;</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="n">  file_permission</span> <span class="o">=</span> <span class="s2">&#34;0644&#34;</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="n">  content</span> <span class="o">=</span> <span class="k">yamlencode</span><span class="p">(</span>{
</span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="n">    all</span> <span class="o">=</span> {
</span></span><span class="line"><span class="ln">21</span><span class="cl"><span class="n">      vars</span> <span class="o">=</span><span class="n"> { ansible_user</span> <span class="o">=</span> <span class="s2">&#34;ops&#34;</span> }
</span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="n">      children</span> <span class="o">=</span> {
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="k">for</span> <span class="k">role</span> <span class="k">in</span> <span class="k">distinct</span><span class="p">(</span><span class="k">flatten</span><span class="p">([</span><span class="k">for</span> <span class="k">h</span> <span class="k">in</span> <span class="k">values</span><span class="p">(</span><span class="k">local</span><span class="p">.</span><span class="k">hosts</span><span class="p">)</span> <span class="err">:</span> <span class="k">h</span><span class="p">.</span><span class="k">roles</span><span class="p">]))</span> <span class="err">:</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="n">        role</span> <span class="o">=</span><span class="n">&gt; { hosts</span> <span class="o">=</span><span class="n"> { for name, h in local.hosts : name</span> <span class="o">=</span><span class="n">&gt; { ansible_host</span> <span class="o">=</span> <span class="k">h</span><span class="p">.</span><span class="k">address</span> } <span class="k">if</span> <span class="k">contains</span><span class="p">(</span><span class="k">h</span><span class="p">.</span><span class="k">roles</span><span class="p">,</span> <span class="k">role</span><span class="p">)</span> } }
</span></span><span class="line"><span class="ln">25</span><span class="cl">      }
</span></span><span class="line"><span class="ln">26</span><span class="cl">    }
</span></span><span class="line"><span class="ln">27</span><span class="cl">  }<span class="p">)</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">}
</span></span><span class="line"><span class="ln">29</span><span class="cl">
</span></span><span class="line"><span class="ln">30</span><span class="cl"><span class="n">output &#34;hosts&#34; { value</span> <span class="o">=</span> <span class="k">local</span><span class="p">.</span><span class="k">hosts</span> }</span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 控制端容器裡</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="nb">cd</span> /work/tf
</span></span><span class="line"><span class="ln">3</span><span class="cl">terraform init
</span></span><span class="line"><span class="ln">4</span><span class="cl">terraform plan
</span></span><span class="line"><span class="ln">5</span><span class="cl">terraform apply -auto-approve
</span></span><span class="line"><span class="ln">6</span><span class="cl">ansible-inventory -i ../ansible/inventory.yml --graph
</span></span><span class="line"><span class="ln">7</span><span class="cl">terraform output -json hosts</span></span></code></pre></div><p><code>required_providers</code> 宣告這份設定要用的 provider（Terraform 操作某一類資源的外掛，<code>hashicorp/local</code> 管本機的檔案），<code>terraform init</code> 依它下載 provider；<code>${path.module}</code> 是 <code>main.tf</code> 所在的目錄，<code>filename</code> 因此指到 <code>tf/</code> 旁邊的 <code>ansible/</code>。<code>children</code> 那兩層 <code>for</code> 把「每台主機有哪些角色」反轉成「每個角色有哪些主機」：外層取出不重複的角色，每個角色成為一個群組，內層把角色串列含有這個角色的主機收進那個群組。<code>apply</code> 的 <code>-auto-approve</code> 跳過執行前的確認提問，這裡用它是因為剛跑過 <code>plan</code> 看過內容。</p>
<p><code>terraform init</code> 印出 <code>Installed hashicorp/local v2.5.2 (signed by HashiCorp)</code>，花了 8.8 秒（下載 provider 的時間，依網路而變）；<a href="/blog/devops/infra/knowledge-cards/terraform-plan-apply/" data-link-title="terraform plan / apply" data-link-desc="IaC 的兩個核心操作：plan 只看不動（產出差異報告）、apply 真的動（執行差異）"><code>plan</code></a> 印 <code>local_file.inventory will be created</code> 與 <code>Plan: 1 to add</code>；<code>apply</code> 之後，<code>--graph</code> 列出 <code>db</code> 群組裡有 <code>mh-both1</code>、<code>web</code> 群組裡有 <code>mh-both1</code> 與 <code>mh-web1</code>，<code>mh-both1</code> 同時在兩個群組裡。產生出來的 <code>inventory.yml</code>（以同一份設定在 <code>hashicorp/terraform:1.9.8</code> 容器裡另外重跑 <code>apply</code> 之後 <code>cat</code> 的輸出；<code>yamlencode</code> 依鍵名排序，字串一律加引號）：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="s2">&#34;all&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w">  </span><span class="s2">&#34;children&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">    </span><span class="s2">&#34;db&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w">      </span><span class="s2">&#34;hosts&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w">        </span><span class="s2">&#34;mh-both1&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">          </span><span class="nt">&#34;ansible_host&#34;: </span><span class="s2">&#34;mh-both1&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w">    </span><span class="s2">&#34;web&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">      </span><span class="s2">&#34;hosts&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">        </span><span class="s2">&#34;mh-both1&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w">          </span><span class="nt">&#34;ansible_host&#34;: </span><span class="s2">&#34;mh-both1&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w">        </span><span class="s2">&#34;mh-web1&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w">          </span><span class="nt">&#34;ansible_host&#34;: </span><span class="s2">&#34;mh-web1&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w">  </span><span class="s2">&#34;vars&#34;</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="w">    </span><span class="nt">&#34;ansible_user&#34;: </span><span class="s2">&#34;ops&#34;</span></span></span></code></pre></div><p><code>address</code> 填的是容器名，Docker network 的內建 DNS 把它解析成容器的位址；真主機上填 <code>&lt;主機資源&gt;.public_ip</code> 這類屬性，或主機在私有網路時的私有位址，看控制端從哪一條網路連進去。</p>
<p><code>terraform output -json hosts</code> 印出同一份主機清單的 JSON：<code>{&quot;mh-both1&quot;:{&quot;address&quot;:&quot;mh-both1&quot;,&quot;roles&quot;:[&quot;web&quot;,&quot;db&quot;]},&quot;mh-web1&quot;:{&quot;address&quot;:&quot;mh-web1&quot;,&quot;roles&quot;:[&quot;web&quot;]}}</code>，給 Ansible 以外的工具讀。</p>
<p><code>local_file</code> 的 <code>file_permission</code> 預設值是 <code>&quot;0777&quot;</code>，實際的檔案權限是它扣掉建立檔案那個程序的 umask。照上面寫了 <code>file_permission = &quot;0644&quot;</code> 的設定做時，<code>inventory.yml</code> 的權限是 <code>644</code>；明寫 <code>0644</code> 讓權限不依賴執行環境的 umask，固定讓其他帳號改不了這份檔案。</p>
<p>這一行是這次第一次 apply 之後才加的。這個控制端容器的 umask 是 <code>0000</code>，所以沒有寫 <code>file_permission</code> 時，<code>stat -c %a ../ansible/inventory.yml</code> 印 <code>777</code>，同一台機器上的任何帳號都改得了 inventory，也就改得了 Ansible 要以 root 連去的主機清單；另一次在 umask 0022 的容器裡重跑是 <code>755</code>。</p>
<p>加上 <code>file_permission = &quot;0644&quot;</code> 之後，<code>plan</code> 印 <code>local_file.inventory must be replaced</code> 與 <code>file_permission = &quot;0777&quot; -&gt; &quot;0644&quot; # forces replacement</code>，apply 之後權限是 <code>644</code>。</p>
<p>產生出來的檔案不手改。下面這個實驗是可以略過的驗證，用來確認手改會被蓋掉：在 <code>inventory.yml</code> 末端加一行 <code># 手加的一行</code> 之後，<code>plan</code> 回報 <code>local_file.inventory will be created</code>，apply 之後那一行消失（<code>grep -c</code> 印 <code>0</code>），跟概念文章寫的結果相同。plan 寫的是重新建立，原因是 <code>local_file</code> 讀取時發現檔案內容跟 State 記的不同，就把這個資源當成不存在（從 plan 的訊息推論的機制）。</p>
<p>手寫 inventory、動態 inventory、直接讀 State 的 inventory plugin 這幾種做法的結果，在〈這一節的其他做法：產生 inventory 的方式〉。</p>
<h3 id="這一節的其他做法產生-inventory-的方式">這一節的其他做法：產生 inventory 的方式</h3>
<p>產生 inventory 的方式與這次的環境下各自的結果：</p>
<ul>
<li><strong>Terraform 的 <code>local_file</code></strong>（這次的做法）：主機清冊只有 <code>locals</code>（真實環境是主機資源）這一個來源，增刪主機時 inventory 跟著改。</li>
<li><strong>手寫 inventory</strong>：不需要 Terraform；主機由 Terraform 建立時，增刪主機要在 Terraform 與 inventory 兩邊各改一次，判斷依據在概念文章〈主機由 Terraform 建立時由它的輸出產生 inventory〉。這次沒有實跑手寫的版本。</li>
<li><strong>Ansible 的動態 inventory</strong>（雲端業者的 inventory plugin 直接查業者的 API）：主機由 autoscaling 這類服務建立、不在 State 裡時也列得出來，代價是控制端要有讀業者 API 的憑證，分組要依主機的標籤（依 Ansible 文件〈Working with dynamic inventory〉與各業者 plugin 的文件，沒有實跑）。</li>
<li><strong>Ansible 的 <code>cloud.terraform</code> collection</strong>：它的 inventory plugin 直接讀 Terraform 的 State 列出主機，不寫 <code>inventory.yml</code> 這份檔案，也就沒有 <code>local_file</code> 的檔案權限與手改被蓋掉這兩件事（依 collection 的文件推論，沒有實跑）。讀 State 與寫檔案兩種做法的判斷依據在概念文章〈主機由 Terraform 建立時由它的輸出產生 inventory〉。</li>
</ul>
<h2 id="變數檔與範本檢查腳本實際要求的欄位兼任兩種角色的主機">變數檔與範本：檢查腳本實際要求的欄位、兼任兩種角色的主機</h2>
<p>變數檔照 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈用 Ansible 的 inventory、群組變數與範本產生每台的清單〉的分層：每台相同的寫在 <code>group_vars/all.yml</code>，依角色不同的寫在角色的檔案，每台不同的寫在 <code>host_vars</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># group_vars/all.yml：每台相同的預設值</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">host_provider</span><span class="p">:</span><span class="w"> </span><span class="l">container</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w"></span><span class="nt">host_region</span><span class="p">:</span><span class="w"> </span><span class="l">local</span><span class="w">
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="w"></span><span class="nt">min_free_bytes</span><span class="p">:</span><span class="w"> </span><span class="m">1073741824</span><span class="w">       </span><span class="c"># 根檔案系統至少留 1 GiB</span><span class="w">
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="w"></span><span class="nt">journal_limit_bytes</span><span class="p">:</span><span class="w"> </span><span class="m">1073741824</span><span class="w">
</span></span></span><span class="line"><span class="ln">6</span><span class="cl"><span class="w"></span><span class="nt">update_timer</span><span class="p">:</span><span class="w"> </span><span class="l">dnf-automatic.timer</span><span class="w">
</span></span></span><span class="line"><span class="ln">7</span><span class="cl"><span class="w"></span><span class="nt">manifest_path</span><span class="p">:</span><span class="w"> </span><span class="l">/etc/host-check/host.toml</span><span class="w">
</span></span></span><span class="line"><span class="ln">8</span><span class="cl"><span class="w"></span><span class="nt">checker_path</span><span class="p">:</span><span class="w"> </span><span class="l">/opt/host-check/host-check.py</span><span class="w">
</span></span></span><span class="line"><span class="ln">9</span><span class="cl"><span class="w"></span><span class="nt">ansible_python_interpreter</span><span class="p">:</span><span class="w"> </span><span class="l">/usr/bin/python3</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># group_vars/web.yml：web 角色；依角色不同的串列用帶角色名的變數名</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">web_entrypoints</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w">  </span>- {<span class="w"> </span><span class="nt">name: http, port: 80, exposure</span><span class="p">:</span><span class="w"> </span><span class="l">cdn-only }</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># group_vars/db.yml：db 角色</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">db_entrypoints</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w">  </span>- {<span class="w"> </span><span class="nt">name: postgres, port: 5432, exposure</span><span class="p">:</span><span class="w"> </span><span class="l">admin-only }</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># host_vars/mh-web1.yml：只有 mh-web1 不同的值</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">min_free_bytes</span><span class="p">:</span><span class="w"> </span><span class="m">2147483648</span><span class="w">       </span><span class="c"># 這台的根檔案系統至少留 2 GiB</span></span></span></code></pre></div><p><code>web.yml</code> 與 <code>db.yml</code> 的入口串列用 <code>web_entrypoints</code>、<code>db_entrypoints</code> 兩個不同的變數名。兩個群組用同一個變數名時，同層的群組依名稱排序合併、後合併的蓋過先合併的，<code>mh-both1</code> 少掉一個角色的入口；合併規則與聯集的寫法在概念文章〈用 Ansible 的 inventory、群組變數與範本產生每台的清單〉。</p>
<p><code>ansible_python_interpreter</code> 指定受管主機上的 Python。沒有指定時 Ansible 自己找，每次執行都印一段 <code>[WARNING]: Platform linux on host mh-web1 is using the discovered Python interpreter at /usr/bin/python3.12</code>，執行結果相同；指定之後警告消失。<code>manifest_path</code> 與 <code>checker_path</code> 是清單與腳本在受管主機上的固定位置，playbook 與執行檢查的指令都用這兩個值。清單放在 <code>/etc/host-check/</code>，因為 <code>/etc</code> 是 FHS 規定放設定檔的位置，主機備份設定檔時會一起收進去。<code>host_provider</code> 與 <code>host_region</code> 寫進清單 <code>[host]</code> 段的 <code>provider</code> 與 <code>region</code>，這次填 <code>container</code> 與 <code>local</code>，真主機填業者與區域的名稱；主機分在幾家業者或幾個區域時，這兩個值就從每台相同改成依群組或每台不同。</p>
<p>檢查腳本載入清單時逐欄比對型別與允許的值，所以範本照腳本要求的欄位寫。<code>host-check.py</code> 要求的欄位：</p>
<ul>
<li><strong><code>[host]</code> 要有 <code>name</code>、<code>provider</code>、<code>region</code>、<code>role</code> 四個字串</strong>：<code>role</code> 只收一個字串，兼任兩種角色的主機寫成 <code>group_names | join(',')</code>，<code>mh-both1</code> 得到 <code>&quot;db,web&quot;</code>。<code>group_names</code> 是 Ansible 替每台主機算出的所屬群組清單（不含 <code>all</code>），依群組名排序；<code>name</code> 用的 <code>inventory_hostname</code> 是那台主機在 inventory 裡的名字（<code>mh-web1</code>、<code>mh-both1</code>）。</li>
<li><strong><code>[listen_audit]</code> 是 <code>protocols</code> 與 <code>allow</code></strong>：允許接聽的連接埠由兩處合起來，<code>where = &quot;host&quot;</code> 的每一筆 <code>[[entrypoint]]</code> 自動算進去，<code>allow</code> 只寫入口以外、刻意允許的連接埠（每一筆是 <code>{ port, protocol, reason }</code>）。所以入口的串列取聯集之後，<code>listen_audit</code> 就跟著有了兩個角色的連接埠，<code>allow</code> 這次是空的。</li>
<li><strong><code>[[filesystem]]</code> 的門檻是 <code>min_free_bytes</code></strong>：剩餘空間的位元組數下限。</li>
<li><strong><code>exposure</code> 只能是 <code>public</code>、<code>cdn-only</code>、<code>admin-only</code>、<code>loopback</code></strong>：寫了清單外的值（例如 <code>private</code>）時，檢查腳本把清單判成格式錯（exit code 2），〈一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc〉實測過。</li>
<li><strong><code>[[entrypoint]]</code> 要有 <code>protocol</code>、<code>where</code>、<code>enforced_by</code>，每一層防護至少一筆 <code>[[entrypoint.verify]]</code></strong>：<code>enforced_by</code> 列出這個入口靠哪幾種防護限制來源（雲端防火牆、主機防火牆這類），每一種是一「層」，每一層至少寫一筆 <code>[[entrypoint.verify]]</code> 說明怎麼確認那一層（同一層寫兩筆也可以，這次每層寫一筆）。<code>where</code> 寫在哪裡要確認：<code>host</code> 是檢查腳本在主機上自己確認，<code>console</code> 是要人到業者主控台確認。這次每個入口寫一層 <code>cloud-firewall</code>，<code>where = &quot;console&quot;</code>，執行時印成要到主控台確認的 <code>MANUAL</code>。</li>
<li><strong><code>[[log]]</code>、<code>[[accumulation]]</code>、<code>[updates]</code>、<code>[[suspend]]</code>、<code>[[resume]]</code> 都是必填</strong>：少了任何一段，腳本判成格式錯。這次各寫一筆最小的內容。<code>[[suspend]]</code> 與 <code>[[resume]]</code> 的每一步要寫出主機停下來時還計不計費（<code>billed_when_stopped</code> 這幾個欄位）；容器沒有計費，填 <code>unknown</code>，欄位的意思在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈清單欄位的設計：檢查腳本要讀的與人要讀的〉。</li>
</ul>
<p>跟概念文章那份範本對照：概念文章的範本是示範值怎麼代入的節錄，鍵名有幾個是那篇自己取的，例如它的 <code>[[filesystem]]</code> 用 <code>warn_percent</code>（使用率的百分比），檢查腳本不認得這個鍵。</p>
<p>下面是 <code>ansible/host.toml.j2</code> 的全文。字串值一律經過 <code>to_json</code> 輸出，引號與反斜線由它跳脫，含中文的值加 <code>ensure_ascii=False</code> 讓中文原樣寫出；直接輸出時會發生什麼，在概念文章〈值寫進 TOML 的方式：直接輸出、to_json 與 ensure_ascii〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="x"># 由 Ansible 依 host.toml.j2 產生，不在主機上手改。TOML 1.0。
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="x">[vars]
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="x">journal_dir = &#34;/var/log/journal&#34;
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="x">[host]
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="x">name = </span><span class="cp">{{</span> <span class="nv">inventory_hostname</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="x">provider = </span><span class="cp">{{</span> <span class="nv">host_provider</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="x">region = </span><span class="cp">{{</span> <span class="nv">host_region</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="x">role = </span><span class="cp">{{</span> <span class="nv">group_names</span> <span class="o">|</span> <span class="nf">join</span><span class="o">(</span><span class="s1">&#39;,&#39;</span><span class="o">)</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="x">[listen_audit]
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="x">protocols = [&#34;tcp&#34;]
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="x">allow = []
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="x"></span><span class="cp">{%</span> <span class="k">set</span> <span class="nv">eps</span> <span class="o">=</span> <span class="o">[{</span><span class="s1">&#39;name&#39;</span><span class="o">:</span> <span class="s1">&#39;ssh&#39;</span><span class="o">,</span> <span class="s1">&#39;port&#39;</span><span class="o">:</span> <span class="m">22</span><span class="o">,</span> <span class="s1">&#39;exposure&#39;</span><span class="o">:</span> <span class="s1">&#39;admin-only&#39;</span><span class="o">}]</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">   <span class="o">+</span> <span class="o">(</span><span class="nv">web_entrypoints</span> <span class="k">if</span> <span class="s1">&#39;web&#39;</span> <span class="k">in</span> <span class="nv">group_names</span> <span class="k">else</span> <span class="o">[])</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">   <span class="o">+</span> <span class="o">(</span><span class="nv">db_entrypoints</span> <span class="k">if</span> <span class="s1">&#39;db&#39;</span> <span class="k">in</span> <span class="nv">group_names</span> <span class="k">else</span> <span class="o">[])</span> <span class="cp">%}</span><span class="x">
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="x"></span><span class="cp">{%</span> <span class="k">for</span> <span class="nv">ep</span> <span class="k">in</span> <span class="nv">eps</span> <span class="cp">%}</span><span class="x">
</span></span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="x">[[entrypoint]]
</span></span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="x">name = </span><span class="cp">{{</span> <span class="nv">ep.name</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">21</span><span class="cl"><span class="x">port = </span><span class="cp">{{</span> <span class="nv">ep.port</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="x">protocol = &#34;tcp&#34;
</span></span></span><span class="line"><span class="ln">23</span><span class="cl"><span class="x">exposure = </span><span class="cp">{{</span> <span class="nv">ep.exposure</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="x">where = &#34;host&#34;
</span></span></span><span class="line"><span class="ln">25</span><span class="cl"><span class="x">enforced_by = [&#34;cloud-firewall&#34;]
</span></span></span><span class="line"><span class="ln">26</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">27</span><span class="cl"><span class="x">  [[entrypoint.verify]]
</span></span></span><span class="line"><span class="ln">28</span><span class="cl"><span class="x">  layer = &#34;cloud-firewall&#34;
</span></span></span><span class="line"><span class="ln">29</span><span class="cl"><span class="x">  where = &#34;console&#34;
</span></span></span><span class="line"><span class="ln">30</span><span class="cl"><span class="x">  console_path = &#34;業者主控台 → 網路 → 防火牆規則&#34;
</span></span></span><span class="line"><span class="ln">31</span><span class="cl"><span class="x">  expect = </span><span class="cp">{{</span> <span class="o">(</span><span class="s2">&#34;TCP &#34;</span> <span class="o">~</span> <span class="nv">ep.port</span> <span class="o">~</span> <span class="s2">&#34; 的規則只對 &#34;</span> <span class="o">~</span> <span class="nv">ep.exposure</span> <span class="o">~</span> <span class="s2">&#34; 的來源開放&#34;</span><span class="o">)</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">32</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">33</span><span class="cl"><span class="x"></span><span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span><span class="x">
</span></span></span><span class="line"><span class="ln">34</span><span class="cl"><span class="x">[[log]]
</span></span></span><span class="line"><span class="ln">35</span><span class="cl"><span class="x">name = &#34;sshd&#34;
</span></span></span><span class="line"><span class="ln">36</span><span class="cl"><span class="x">source = &#34;journal:sshd.service&#34;
</span></span></span><span class="line"><span class="ln">37</span><span class="cl"><span class="x">look_for = &#34;登入失敗與不認得的帳號&#34;
</span></span></span><span class="line"><span class="ln">38</span><span class="cl"><span class="x">interval_days = 7
</span></span></span><span class="line"><span class="ln">39</span><span class="cl"><span class="x">manual_command = &#34;journalctl -u sshd --since -7d&#34;
</span></span></span><span class="line"><span class="ln">40</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">41</span><span class="cl"><span class="x">[[accumulation]]
</span></span></span><span class="line"><span class="ln">42</span><span class="cl"><span class="x">name = &#34;journal&#34;
</span></span></span><span class="line"><span class="ln">43</span><span class="cl"><span class="x">path = [&#34;</span><span class="cp">{%</span> <span class="k">raw</span> <span class="cp">%}</span>{{journal_dir}}<span class="cp">{%</span> <span class="k">endraw</span> <span class="cp">%}</span><span class="x">&#34;]
</span></span></span><span class="line"><span class="ln">44</span><span class="cl"><span class="x">missing_ok = true
</span></span></span><span class="line"><span class="ln">45</span><span class="cl"><span class="x">limit_bytes = </span><span class="cp">{{</span> <span class="nv">journal_limit_bytes</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">46</span><span class="cl"><span class="x">purge_command = &#34;journalctl --vacuum-size=500M&#34;
</span></span></span><span class="line"><span class="ln">47</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">48</span><span class="cl"><span class="x">[[filesystem]]
</span></span></span><span class="line"><span class="ln">49</span><span class="cl"><span class="x">name = &#34;root&#34;
</span></span></span><span class="line"><span class="ln">50</span><span class="cl"><span class="x">mount = &#34;/&#34;
</span></span></span><span class="line"><span class="ln">51</span><span class="cl"><span class="x">min_free_bytes = </span><span class="cp">{{</span> <span class="nv">min_free_bytes</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">52</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">53</span><span class="cl"><span class="x">[updates]
</span></span></span><span class="line"><span class="ln">54</span><span class="cl"><span class="x">mode = &#34;download-only&#34;
</span></span></span><span class="line"><span class="ln">55</span><span class="cl"><span class="x">timer = </span><span class="cp">{{</span> <span class="nv">update_timer</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">56</span><span class="cl"><span class="x">config_files = [&#34;/etc/dnf/automatic.conf&#34;]
</span></span></span><span class="line"><span class="ln">57</span><span class="cl"><span class="x">expect = { download_updates = &#34;yes&#34;, apply_updates = &#34;no&#34; }
</span></span></span><span class="line"><span class="ln">58</span><span class="cl"><span class="x">reboot_check = &#34;dnf needs-restarting -r&#34;
</span></span></span><span class="line"><span class="ln">59</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">60</span><span class="cl"><span class="x">[[suspend]]
</span></span></span><span class="line"><span class="ln">61</span><span class="cl"><span class="x">step = 1
</span></span></span><span class="line"><span class="ln">62</span><span class="cl"><span class="x">resource = &#34;主機&#34;
</span></span></span><span class="line"><span class="ln">63</span><span class="cl"><span class="x">billed_when_stopped = &#34;unknown&#34;
</span></span></span><span class="line"><span class="ln">64</span><span class="cl"><span class="x">billed_when_stopped_beyond_free = &#34;unknown&#34;
</span></span></span><span class="line"><span class="ln">65</span><span class="cl"><span class="x">billing_basis = &#34;依業者的計費文件&#34;
</span></span></span><span class="line"><span class="ln">66</span><span class="cl"><span class="x">where = &#34;console&#34;
</span></span></span><span class="line"><span class="ln">67</span><span class="cl"><span class="x">console_path = &#34;業者主控台 → 主機 → 停止&#34;
</span></span></span><span class="line"><span class="ln">68</span><span class="cl"><span class="x">verify = &#34;主機的狀態是已停止&#34;
</span></span></span><span class="line"><span class="ln">69</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">70</span><span class="cl"><span class="x">[[resume]]
</span></span></span><span class="line"><span class="ln">71</span><span class="cl"><span class="x">step = 1
</span></span></span><span class="line"><span class="ln">72</span><span class="cl"><span class="x">resource = &#34;主機&#34;
</span></span></span><span class="line"><span class="ln">73</span><span class="cl"><span class="x">billed_when_stopped = &#34;unknown&#34;
</span></span></span><span class="line"><span class="ln">74</span><span class="cl"><span class="x">billed_when_stopped_beyond_free = &#34;unknown&#34;
</span></span></span><span class="line"><span class="ln">75</span><span class="cl"><span class="x">billing_basis = &#34;依業者的計費文件&#34;
</span></span></span><span class="line"><span class="ln">76</span><span class="cl"><span class="x">where = &#34;console&#34;
</span></span></span><span class="line"><span class="ln">77</span><span class="cl"><span class="x">console_path = &#34;業者主控台 → 主機 → 啟動&#34;
</span></span></span><span class="line"><span class="ln">78</span><span class="cl"><span class="x">verify = &#34;主機的狀態是執行中&#34;</span></span></span></code></pre></div><p>範本裡每台相同的值（ssh 的入口、log、累積的資料、更新的設定、暫停與恢復的步驟）直接寫在範本本身：這些值在所有主機都相同、也不隨角色變，放進 <code>group_vars/all.yml</code> 只多一層間接；它開始依群組或主機而變時，才換成變數。依角色不同的入口由 <code>eps</code> 那一行依 <code>group_names</code> 取聯集，每台不同的只有主機名與 <code>min_free_bytes</code>。<code>[[accumulation]]</code> 的 <code>path</code> 用檢查腳本自己的替換語法 <code>{{journal_dir}}</code>，包在 <code>{% raw %}</code> 與 <code>{% endraw %}</code> 之間，Jinja2 原樣輸出，留給檢查腳本換成 <code>[vars]</code> 的值；理由在概念文章〈範本的 {{ }} 與清單自己的替換語法同形〉。</p>
<p>部署到主機之前，先在控制端用 <code>-c local</code> 產生一份到 <code>out/</code> 看內容：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># 控制端容器裡，在 /work/ansible 執行；out/ 要先存在</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">mkdir -p out
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">ansible all -i inventory.yml -c <span class="nb">local</span> -e <span class="nv">ansible_python_interpreter</span><span class="o">=</span>/usr/local/bin/python3 <span class="se">\
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="se"></span>  -m template -a <span class="s2">&#34;src=host.toml.j2 dest=out/{{ inventory_hostname }}.toml&#34;</span> -o
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">python3 -c <span class="s1">&#39;
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="s1">import sys, tomllib
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="s1">for f in sys.argv[1:]:
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="s1">    d = tomllib.load(open(f, &#34;rb&#34;))
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="s1">    print(f, d[&#34;host&#34;][&#34;role&#34;], [(e[&#34;name&#34;], e[&#34;port&#34;]) for e in d[&#34;entrypoint&#34;]], d[&#34;filesystem&#34;][0][&#34;min_free_bytes&#34;], d[&#34;accumulation&#34;][0][&#34;path&#34;])
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="s1">&#39;</span> out/*.toml</span></span></code></pre></div><p><code>ansible</code> 這一條兩台都印 <code>CHANGED</code>，exit code 0。<code>-m</code> 指定要執行的模組（這裡是 <code>template</code>），<code>-a</code> 是給那個模組的參數，<code>-o</code> 把每台的結果擠成一行印出。<code>-c local</code> 讓 Ansible 在控制端自己身上執行模組，這時「受管主機」就是控制端，而 <code>group_vars/all.yml</code> 的 <code>ansible_python_interpreter</code>（<code>/usr/bin/python3</code>）照樣套用；<code>python:3.12-slim</code> 沒有 <code>/usr/bin/python3</code>，Python 裝在 <code>/usr/local/bin/python3</code>，所以用 <code>-e</code> 換成控制端的路徑，不換的話這一條以 exit code 2 失敗。<code>python3</code> 那一段用 <code>tomllib</code> 讀回兩份清單，印出：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="ln">1</span><span class="cl">out/mh-both1.toml db,web [(&#39;ssh&#39;, 22), (&#39;http&#39;, 80), (&#39;postgres&#39;, 5432)] 1073741824 [&#39;{{journal_dir}}&#39;]
</span></span><span class="line"><span class="ln">2</span><span class="cl">out/mh-web1.toml web [(&#39;ssh&#39;, 22), (&#39;http&#39;, 80)] 2147483648 [&#39;{{journal_dir}}&#39;]</span></span></code></pre></div><p><code>mh-both1</code> 有兩個角色的入口，門檻沿用 <code>all</code> 的 1 GiB；<code>mh-web1</code> 只有 web 的入口，門檻是 <code>host_vars</code> 的 2 GiB；兩份的 <code>path</code> 都留著 <code>{{journal_dir}}</code>。<code>tomllib</code> 讀得懂只代表語法正確；要確認欄位是否符合檢查腳本的規格，得讓檢查腳本自己載入一次，〈部署清單：建目錄、copy 與 template、validate 改用檢查腳本的載入函式、&ndash;check 的預覽〉裡 template 任務的 <code>validate</code> 做的就是這件事。<code>out/</code> 的檔案在這個 umask <code>0000</code> 的控制端上寫成 <code>0666</code>；它們只用來預覽，看完刪掉即可。省略這一步時的結果，在〈這一節的其他做法：省略本機的預覽〉。</p>
<h3 id="這一節的其他做法省略本機的預覽">這一節的其他做法：省略本機的預覽</h3>
<p>這一步可以省略，直接部署也會寫出同樣的內容；先在本機產生一份的好處是不連任何主機就看得到每台的清單，改範本時可以先比對 <code>out/</code> 裡新舊兩份的差異。</p>
<h2 id="部署清單建目錄copy-與-templatevalidate-改用檢查腳本的載入函式check-的預覽">部署清單：建目錄、copy 與 template、validate 改用檢查腳本的載入函式、&ndash;check 的預覽</h2>
<p>部署有三個任務：建放腳本與清單的目錄、把檢查腳本複製過去、依範本寫出每台的清單。三個任務都要以 root 寫進 <code>/opt</code> 與 <code>/etc</code>，所以整個 play 設 <code>become: true</code>，Ansible 以 <code>ops</code> 登入後經由 sudo 切成 root。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c"># deploy.yml</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w"></span>- <span class="nt">hosts</span><span class="p">:</span><span class="w"> </span><span class="l">all</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">  </span><span class="nt">become</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w">  </span><span class="nt">tasks</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">放腳本與清單的目錄</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">      </span><span class="nt">ansible.builtin.file</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w">        </span><span class="nt">path</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ item }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">        </span><span class="nt">state</span><span class="p">:</span><span class="w"> </span><span class="l">directory</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">        </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w">        </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;0755&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="w">      </span><span class="nt">loop</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="w">        </span>- <span class="s2">&#34;{{ checker_path | dirname }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="w">        </span>- <span class="s2">&#34;{{ manifest_path | dirname }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">檢查腳本</span><span class="w">
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="w">      </span><span class="nt">ansible.builtin.copy</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="w">        </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="l">files/host-check.py</span><span class="w">
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ checker_path }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="w">        </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="w">        </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="ln">21</span><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;0755&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="w">    </span>- <span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l">每台的清單（寫出之前用檢查腳本自己的載入函式驗證）</span><span class="w">
</span></span></span><span class="line"><span class="ln">23</span><span class="cl"><span class="w">      </span><span class="nt">ansible.builtin.template</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="w">        </span><span class="nt">src</span><span class="p">:</span><span class="w"> </span><span class="l">host.toml.j2</span><span class="w">
</span></span></span><span class="line"><span class="ln">25</span><span class="cl"><span class="w">        </span><span class="nt">dest</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;{{ manifest_path }}&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">26</span><span class="cl"><span class="w">        </span><span class="nt">owner</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="ln">27</span><span class="cl"><span class="w">        </span><span class="nt">group</span><span class="p">:</span><span class="w"> </span><span class="l">root</span><span class="w">
</span></span></span><span class="line"><span class="ln">28</span><span class="cl"><span class="w">        </span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="s2">&#34;0644&#34;</span><span class="w">
</span></span></span><span class="line"><span class="ln">29</span><span class="cl"><span class="w">        </span><span class="nt">validate</span><span class="p">:</span><span class="w"> </span><span class="p">&gt;-</span><span class="sd">
</span></span></span><span class="line"><span class="ln">30</span><span class="cl"><span class="sd">          python3 -I -c &#39;import runpy, sys;
</span></span></span><span class="line"><span class="ln">31</span><span class="cl"><span class="sd">          runpy.run_path(sys.argv[1], run_name=&#34;hc&#34;)[&#34;load_manifest&#34;](sys.argv[2])&#39;
</span></span></span><span class="line"><span class="ln">32</span><span class="cl"><span class="sd">          {{ checker_path }} %s</span></span></span></code></pre></div><p>建目錄的任務用 <code>loop</code> 對串列裡的每一項各執行一次，每一次的值放在 <code>item</code>；<code>dirname</code> 過濾器取路徑的上層目錄，<code>/opt/host-check/host-check.py</code> 得到 <code>/opt/host-check</code>。<code>validate: &gt;-</code> 是 YAML 的折疊寫法：底下幾行接成一行、行與行之間換成空白、結尾不留換行，長指令可以分行寫。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 控制端容器裡</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible-playbook -i inventory.yml deploy.yml --check --diff   <span class="c1"># 預覽，不寫任何檔案</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl">ansible-playbook -i inventory.yml deploy.yml</span></span></code></pre></div><p>加上 <code>file</code> 任務之後，第一次執行三個任務都是 <code>changed</code>，再執行一次 <code>changed=0</code>，花 1.9 秒（兩台，控制端與主機在同一台開發機上，真主機要加上網路來回的時間）。</p>
<p><strong>這次遇到的錯誤</strong>：建目錄的任務是第一次實跑失敗之後加的。最初的 playbook 只有 copy 與 template 兩個任務，<code>--check --diff</code> 預覽時兩台都回報 <code>changed=2</code>、<code>failed=0</code>；實際執行時兩台都在 copy 那一步失敗，訊息是 <code>Destination directory /opt/host-check does not exist</code>，exit code 2。copy 與 template 都不建上層目錄（template 單獨執行時同樣以 <code>Destination directory … does not exist</code> 失敗，另外單獨實跑過一次確認），而 check 模式沒有實際寫檔，碰不到這個錯誤，所以預覽的 <code>failed=0</code> 不代表實際執行會成功。</p>
<p><strong><code>validate</code> 用的是檢查腳本自己的載入函式</strong>。<a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈產生時驗證清單與比對主機上的清單〉的 <code>validate</code> 範例只用 <code>tomllib</code> 解析，驗得到語法、驗不到欄位。<code>host-check.py</code> 的 <code>load_manifest()</code> 讀清單並逐欄驗證（逐欄驗證的程式在原始碼那一篇〈清單的載入與驗證：Probe、_Validator、load_manifest 與 validate〉），格式錯時丟出 <code>ManifestError</code>，不執行任何檢查，所以 <code>validate</code> 用 <code>runpy.run_path</code> 載入腳本、只呼叫這個函式：函式丟出例外時 Python 以 exit 1 結束，Ansible 判成驗證失敗，那一台的清單維持原樣。</p>
<ul>
<li><strong>用 <code>runpy.run_path</code> 載入腳本</strong>：腳本開頭有 <code>from __future__ import annotations</code>，<code>dataclass</code> 要從 <code>sys.modules</code> 找到這個模組。<code>runpy.run_path</code> 執行期間會把它登記進去，<code>run_name=&quot;hc&quot;</code> 讓腳本結尾的 <code>if __name__ == &quot;__main__&quot;</code> 不成立，<code>main()</code> 不會跑。</li>
<li><strong><code>%s</code> 與 <code>{{ checker_path }}</code></strong>：<code>%s</code> 由 Ansible 換成暫存檔的路徑，<code>{{ checker_path }}</code> 換成上一個任務放好的腳本。<code>validate</code> 在受管主機上執行，用的是主機上的 Python 與主機上那一份腳本，所以 copy 的任務要排在 template 之前，主機上的 Python 也要是 3.11 以上（<code>tomllib</code>）。</li>
<li><strong><code>-I</code></strong>：Python 的隔離模式，忽略所有 <code>PYTHON*</code> 環境變數，不把目前目錄與使用者的 site-packages 加進模組的搜尋路徑；執行 <code>validate</code> 時的工作目錄裡放了 <code>json.py</code> 這類跟標準函式庫同名的檔案，也不會取代標準函式庫被載入。</li>
</ul>
<p><strong>這次遇到的錯誤</strong>：寫進 <code>validate</code> 之前，先在控制端拿 <code>out/</code> 的兩份清單試載入；最初用 <code>importlib.util.spec_from_file_location</code> 載入，在腳本的 <code>@dataclass</code> 那一行丟出 <code>AttributeError: 'NoneType' object has no attribute '__dict__'</code>，因為那樣載入的模組沒有登記進 <code>sys.modules</code>。</p>
<p><strong>check 模式不執行 <code>validate</code></strong>：另外在一個裝 ansible-core 2.17.7 的容器裡，以 <code>-c local</code> 對一個 template 任務設 <code>validate</code> 指向一條一定 exit 1、執行時會留下紀錄的指令，<code>--check --diff</code> 回報 <code>changed</code>、exit code 0，紀錄檔沒有出現；拿掉 <code>--check</code> 實際執行時以 <code>failed to validate</code> 失敗、exit code 2，紀錄檔有那一行。所以會被 <code>validate</code> 擋下的清單，<code>--check</code> 的預覽照樣顯示成功。</p>
<p><code>--check --diff</code> 對 copy 任務會印出整份腳本的差異（這次一千多行），清單的差異埋在後面；只想看清單時，用〈一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc〉那一節的 ad-hoc 指令，只對 template 模組跑 <code>--check --diff</code>。</p>
<p><code>validate</code> 的其他寫法、逐台驗證造成的部分部署、copy 任務的替代，在〈這一節的其他做法：validate 的寫法、部分部署與 copy 的替代〉。</p>
<h3 id="這一節的其他做法validate-的寫法部分部署與-copy-的替代">這一節的其他做法：validate 的寫法、部分部署與 copy 的替代</h3>
<p>這個 <code>validate</code> 擋得下的範圍在下面〈一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc〉實測過一次：db 角色寫了 <code>exposure: private</code> 時，只有 <code>mh-both1</code> 驗證失敗、維持舊的清單，<code>mh-web1</code> 照常部署。三種寫法的差別：</p>
<ul>
<li><strong>檢查腳本的載入函式</strong>（這次的做法）：型別、允許的值、必填欄位、白名單都在寫出之前驗過，主機上不會出現一份腳本拒絕載入的清單。需要腳本有這樣的函式，而且 <code>validate</code> 依賴腳本內部的函式名，腳本改名時這一行要跟著改。</li>
<li><strong>只用 <code>tomllib</code></strong>：只要求主機上有 Python 3.11 以上，不依賴腳本的內部；<code>exposure = &quot;private&quot;</code> 這種語法正確、值不允許的寫法會被寫到主機上，等下一次執行檢查時腳本以 exit code 2 結束（<code>exposure = &quot;private&quot;</code> 是合法的 TOML，所以 <code>tomllib</code> 照常解析，這一點是從 TOML 的語法推的，沒有另外實跑）。</li>
<li><strong>不寫 <code>validate</code></strong>：寫壞的清單直接上主機。〈一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc〉那一節在主機上手改清單的實驗就是這個結果：那台主機在修好之前一項檢查都沒有跑。</li>
</ul>
<p><code>validate</code> 逐台執行，一台驗證失敗時其他台照常寫出新版，結果是部分部署：這次 <code>exposure: private</code> 那一輪，<code>mh-web1</code> 照常部署、<code>mh-both1</code> 留在舊版；範本的改動同時涉及兩台時，兩台就會停在不同的版本。要全部主機一起成功或一起不動時，有兩種替代：先在控制端用 <code>-c local</code> 產生全部主機的清單，用同一個載入函式驗完再部署；或在 play 設 <code>any_errors_fatal: true</code>，有一台失敗就停下整個 play（兩種都是推論，沒有實跑；<code>any_errors_fatal</code> 停的是之後的任務，失敗那一刻之前已經寫出的主機不會退回）。</p>
<p>copy 那一個任務另有兩種做法：把檢查腳本放進主機的映像檔，或打包成套件由套件管理員安裝。兩種都讓腳本的版本跟著映像檔或套件的版本走，代價是改腳本要重建映像檔或重新打包；打包成 rpm 時，另外換到 <code>rpm -V</code> 查得出主機上那一份腳本有沒有被改過。這次腳本與清單一起由 Ansible 部署，改腳本與改範本是同一個步驟。這一點是從部署的流程推的，沒有實作另外兩種。</p>
<h2 id="以--b-與-systemd-run-執行檢查腳本每台的輸出與-ansible-的-exit-code">以 -b 與 systemd-run 執行檢查腳本：每台的輸出與 ansible 的 exit code</h2>
<p>執行檢查之前，先確認 <code>systemd-run</code> 的限制在這次的容器裡有沒有套上：<a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a> 記錄過 OrbStack 的 LXC 機器會讓限制靜默無效。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 控制端容器裡</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible all -i inventory.yml -b -m <span class="nb">command</span> -a <span class="s2">&#34;systemd-detect-virt&#34;</span> -o
</span></span><span class="line"><span class="ln">3</span><span class="cl">ansible mh-web1 -i inventory.yml -b -m <span class="nb">command</span> -a <span class="s2">&#34;systemd-run --wait --pipe --collect -p ProtectSystem=strict touch /var/x&#34;</span></span></span></code></pre></div><p><code>systemd-detect-virt</code> 兩台都印 <code>docker</code>（Docker 容器；LXC 環境印的是 <code>lxc</code>）；<code>touch /var/x</code> 那一條在 <code>mh-web1</code> 上以 <code>rc=1</code> 失敗，輸出 <code>/bin/touch: cannot touch '/var/x': Read-only file system</code>，<code>ProtectSystem=strict</code> 套上了。印的是 <code>lxc</code>、或 <code>touch</code> 成功時，這個環境量不到沙箱的效果，真主機上的確認方式在〈容器跟真主機不同的地方，以及在真主機上要另外確認的項目〉。這一條的 <code>ansible</code> 指令本身以 exit code 2 結束，是預期的結果：<code>rc=1</code> 正是要看到的失敗（整批的 exit code 怎麼讀，在 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈用 Ansible 對多台主機執行檢查腳本：整批的 exit code、每台的 rc 與執行身分的代價〉）。</p>
<p>執行檢查的指令寫成一支小腳本，屬性照 TOML 那一篇〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉的 <code>systemd-run</code> 寫法。這次量輸出時 <code>check.sh</code> 寫的是固定路徑 <code>/opt/host-check/host-check.py</code> 與 <code>/etc/host-check/host.toml</code>，跟下面用變數的版本代入之後的值相同：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="cp">#!/bin/bash
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="cp"></span><span class="c1"># check.sh：對 inventory 的每台主機以 root、在 systemd 沙箱裡執行檢查腳本；&#34;$@&#34; 接其他 ansible 參數（例如 --limit）</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl">ansible all -i inventory.yml -b -m <span class="nb">command</span> -o -a <span class="s2">&#34;systemd-run --wait --pipe --collect \
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="s2">  -p ProtectSystem=strict -p &#39;ReadWritePaths=/var/log /var/cache/dnf&#39; \
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="s2">  -p StateDirectory=host-check -p StateDirectoryMode=0700 \
</span></span></span><span class="line"><span class="ln">6</span><span class="cl"><span class="s2">  -p NoNewPrivileges=yes -p PrivateTmp=yes -p IPAddressDeny=any \
</span></span></span><span class="line"><span class="ln">7</span><span class="cl"><span class="s2">  python3 {{ checker_path }} --manifest {{ manifest_path }}&#34;</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span></span></span></code></pre></div><p><code>check.sh</code> 建在控制端的 <code>/work/ansible</code>（開發機上是工作目錄的 <code>ansible/</code>），建好之後在控制端容器裡加上執行權限，下文以 <code>./check.sh</code> 執行：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl">chmod +x check.sh</span></span></code></pre></div><p>要依環境換的位置只有 <code>all</code>（要檢查的主機範圍）。腳本與清單的路徑寫成 <code>{{ checker_path }}</code> 與 <code>{{ manifest_path }}</code>，Ansible 執行時用每台主機的變數代入，跟部署時用的是同一組值；寫死路徑的話，<code>group_vars</code> 改了位置而 <code>check.sh</code> 沒改時，檢查會安靜地讀舊路徑上過期的清單。<code>command</code> 模組用 <code>shlex</code> 切參數、不經過 shell，所以 <code>'ReadWritePaths=/var/log /var/cache/dnf'</code> 的單引號讓兩個路徑留在同一個參數裡。變數代入與引號的切法另外在一個裝 ansible-core 2.17.7 的容器裡，以 <code>-c local</code> 對設了這兩個變數的 localhost 實跑過一次：<code>{{ checker_path }}</code> 與 <code>{{ manifest_path }}</code> 換成了變數的值，<code>'ReadWritePaths=/var/log /var/cache/dnf'</code> 留在同一個參數裡。不加 <code>--manifest</code> 時，腳本讀它自己所在目錄的 <code>host.toml</code>，這次清單放在 <code>/etc/host-check/</code>，所以要寫。</p>
<p>讀下面的輸出之前，先把兩個 exit code 分開：<code>ansible</code> 指令本身的 exit code 是整批的結果，每台的 <code>rc</code> 是那台主機上檢查腳本的 exit code。</p>
<table>
  <thead>
      <tr>
          <th><code>ansible</code> 指令的 exit code（整批）</th>
          <th>代表</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>0</td>
          <td>每台的任務都成功</td>
      </tr>
      <tr>
          <td>2</td>
          <td>至少一台主機的任務失敗（標成 <code>FAILED!</code>）</td>
      </tr>
      <tr>
          <td>4</td>
          <td>有主機連不上（標成 <code>UNREACHABLE!</code>）</td>
      </tr>
  </tbody>
</table>
<table>
  <thead>
      <tr>
          <th>每台的 <code>rc</code>（檢查腳本的 exit code）</th>
          <th>代表</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>0</td>
          <td>沒有 <code>WARN</code></td>
      </tr>
      <tr>
          <td>1</td>
          <td>至少一項 <code>WARN</code></td>
      </tr>
      <tr>
          <td>2</td>
          <td>清單格式錯，或命令列參數錯（stderr 印用法說明而不是 <code>manifest error:</code>）</td>
      </tr>
  </tbody>
</table>
<p>讀的順序：先看整批的值決定要不要告警，再看每台的 <code>rc</code> 決定處置；<code>rc</code> 是 1 時另看輸出裡有沒有 <code>summary:</code> 那一行。</p>
<p><code>./check.sh</code> 第一次執行，<code>ansible</code> 的 exit code 是 0，花 0.9 秒。兩台都印 <code>CHANGED | rc=0</code>；<code>command</code> 模組不論指令做了什麼一律回報 changed，所以這個 <code>CHANGED</code> 只代表指令執行了，不代表檢查改動了主機。<code>-o</code> 把每台的報告印成一行、換行寫成 <code>\n</code>；要看分行的報告時，拿掉 <code>-o</code> 再跑一次（<code>ansible all -i inventory.yml -b -m command -a &quot;…&quot;</code>，指令照 <code>check.sh</code>），每台的 stdout 會分行印出。<code>mh-both1</code> 的報告：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="ln"> 1</span><span class="cl">PASS   entrypoint/ssh:22/tcp  有程序在聽（0.0.0.0, [::]），exposure=admin-only
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">MANUAL entrypoint/ssh:22/tcp/cloud-firewall  [主控台] 業者主控台 → 網路 → 防火牆規則 → 預期：TCP 22 的規則只對 admin-only 的來源開放
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">PASS   entrypoint/http:80/tcp  有程序在聽（0.0.0.0），exposure=cdn-only
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">MANUAL entrypoint/http:80/tcp/cloud-firewall  [主控台] 業者主控台 → 網路 → 防火牆規則 → 預期：TCP 80 的規則只對 cdn-only 的來源開放
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">PASS   entrypoint/postgres:5432/tcp  有程序在聽（0.0.0.0），exposure=admin-only
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">MANUAL entrypoint/postgres:5432/tcp/cloud-firewall  [主控台] 業者主控台 → 網路 → 防火牆規則 → 預期：TCP 5432 的規則只對 admin-only 的來源開放
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">PASS   listen-audit  loopback 以外只有清單列出的連接埠在聽（tcp）
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">PASS   log/sshd  最新一筆：2026-10-26T13:44:47+00:00 mh-both1 python3[1904]: ansible-ansible.legacy.command Invoked …；每 7 天讀一次：journalctl -u sshd --since -7d
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">PASS   accumulation/journal  路徑不存在，算 0 bytes（/var/log/journal）
</span></span><span class="line"><span class="ln">10</span><span class="cl">PASS   filesystem/root  / 剩 77.7 GiB／共 117.6 GiB（下限 1.0 GiB）
</span></span><span class="line"><span class="ln">11</span><span class="cl">PASS   updates/dnf-automatic.timer/is-enabled  enabled（預期 enabled）
</span></span><span class="line"><span class="ln">12</span><span class="cl">PASS   updates/dnf-automatic.timer/is-active  active（預期 active）
</span></span><span class="line"><span class="ln">13</span><span class="cl">PASS   updates/config  download_updates = yes, apply_updates = no（mode download-only，讀了 /etc/dnf/automatic.conf）
</span></span><span class="line"><span class="ln">14</span><span class="cl">PASS   updates/reboot  不需要重開機
</span></span><span class="line"><span class="ln">15</span><span class="cl">since: 沒有前一次的結果（/var/lib/host-check/last.json）
</span></span><span class="line"><span class="ln">16</span><span class="cl">summary: 11 PASS, 0 WARN, 3 MANUAL（phase maintain，/etc/host-check/host.toml）</span></span></code></pre></div><p><code>mh-web1</code> 少了 postgres 的兩行，結果是 <code>10 PASS, 0 WARN, 2 MANUAL</code>，<code>filesystem/root</code> 的下限印成 <code>2.0 GiB</code>。<code>summary:</code> 裡的 <code>phase maintain</code> 是這次跑的檢查階段：檢查腳本分維護（<code>maintain</code>，預設）、暫停（<code>suspend</code>）與恢復（<code>resume</code>）三種階段，這次跑的是維護。報告之後的 stderr 是 <code>systemd-run</code> 印的那一次執行的資料：<code>Finished with result: success</code>、<code>Service runtime: 378ms</code>、<code>Memory peak: 79M</code>。三個值得讀的位置：</p>
<ul>
<li><strong><code>accumulation/journal</code> 判成 <code>PASS</code> 而路徑不存在</strong>：<code>[[accumulation]]</code> 寫了 <code>missing_ok = true</code>，這個容器的 <a href="/blog/linux/dotfile/knowledge-cards/journald/" data-link-title="journald（systemd-journald）" data-link-desc="用 journalctl 查不到重開機之前的紀錄、journal 佔掉多少磁碟不清楚、某個服務的 log 出現 Suppressed 訊息，或要判斷一筆紀錄該去 journal 還是 /var/log 底下的檔案找時">journald（systemd-journald）</a> 只把 journal 存在記憶體裡（<code>/run/log/journal</code>），<code>/var/log/journal</code> 沒有建出來。<code>path</code> 那一格印出 <code>/var/log/journal</code>，代表檢查腳本把 <code>{{journal_dir}}</code> 換成了 <code>[vars]</code> 的值。</li>
<li><strong><code>log/sshd</code> 的最新一筆是 Ansible 自己留下的</strong>：<code>ansible-ansible.legacy.command Invoked</code> 是這次執行檢查時 Ansible 的模組寫進 journal 的紀錄，而它被算在 <code>sshd.service</code> 底下，所以「journal 裡有 sshd 的紀錄」這一項由檢查的執行本身滿足。成因寫在〈容器跟真主機不同的地方，以及在真主機上要另外確認的項目〉。</li>
<li><strong>結果檔</strong>：<code>/var/lib/host-check</code> 是 <code>700</code>、<code>last.json</code> 是 <code>600</code>，擁有者都是 root，<code>StateDirectoryMode=0700</code> 生效了。第二次執行時 <code>since:</code> 那一行印出上一次的時間，量到的值都跟前一次相同時接著印 <code>，與前一次相同</code>（<code>since: &lt;上一次的時間&gt;，與前一次相同</code>），後面沒有其他行；<code>DELTA &lt;項目&gt; 舊值 → 新值（差）</code> 這種行只在量到的值變了時出現，例如〈一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc〉把門檻改成 1 TiB 的那一輪。</li>
</ul>
<p>不加 <code>-b</code>、不經過 <code>systemd-run</code>、改由每台的 timer 執行、接上排程時讀結果的方式，以及沙箱屬性寫在 <code>check.sh</code> 的代價，在〈這一節的其他做法：-b、systemd-run、執行的位置、排程讀結果與屬性的住址〉。</p>
<h3 id="這一節的其他做法-bsystemd-run執行的位置排程讀結果與屬性的住址">這一節的其他做法：-b、systemd-run、執行的位置、排程讀結果與屬性的住址</h3>
<p>執行檢查這一步，在 <code>-b</code>、<code>systemd-run</code> 與執行的位置上各有替代。「不加 <code>-b</code>」與「不加 <code>-b</code>、也不經過 <code>systemd-run</code>」只對 <code>mh-web1</code> 跑過，指令是：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 控制端容器裡；不加 -b：check.sh 那一條拿掉 -b，只對 mh-web1</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible mh-web1 -i inventory.yml -m <span class="nb">command</span> -o -a <span class="s2">&#34;systemd-run --wait --pipe --collect \
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="s2">  -p ProtectSystem=strict -p &#39;ReadWritePaths=/var/log /var/cache/dnf&#39; \
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="s2">  -p StateDirectory=host-check -p StateDirectoryMode=0700 \
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="s2">  -p NoNewPrivileges=yes -p PrivateTmp=yes -p IPAddressDeny=any \
</span></span></span><span class="line"><span class="ln">6</span><span class="cl"><span class="s2">  python3 /opt/host-check/host-check.py --manifest /etc/host-check/host.toml&#34;</span>
</span></span><span class="line"><span class="ln">7</span><span class="cl"><span class="c1"># 不加 -b、也不經過 systemd-run：以 ops 直接執行檢查腳本</span>
</span></span><span class="line"><span class="ln">8</span><span class="cl">ansible mh-web1 -i inventory.yml -m <span class="nb">command</span> -o -a <span class="s2">&#34;python3 /opt/host-check/host-check.py --manifest /etc/host-check/host.toml --no-state&#34;</span></span></span></code></pre></div><ul>
<li><strong>不加 <code>-b</code></strong>：<code>systemd-run</code> 以 <code>ops</code> 執行，<code>ansible</code> 把 <code>mh-web1</code> 標成 <code>FAILED!</code>、<code>rc</code> 是 1，stderr 是 <code>Failed to start transient service unit: Access denied</code>，檢查腳本沒有執行，<code>ansible</code> 的 exit code 是 2。這個 <code>rc</code> 跟檢查腳本「至少一項 <code>WARN</code>」的 exit code 1 相同，只看 <code>rc</code> 分不開，要看輸出裡有沒有 <code>summary:</code> 那一行：腳本跑完一定印它，沒有這一行代表腳本沒有跑完或沒有開始。</li>
<li><strong>不加 <code>-b</code>、也不經過 <code>systemd-run</code></strong>（以 <code>ops</code> 直接執行檢查腳本；<code>--no-state</code> 不讀也不寫結果檔，這次的測試因此不改到 root 那份 <code>last.json</code>）：腳本照常跑完，stderr 印 <code>note: 不是 root，journal、audit.log 與 collector 的目錄可能讀不到（會印成 WARN）</code>，結果是 <code>9 PASS, 1 WARN, 2 MANUAL</code>，多出來的 <code>WARN</code> 是 <code>log/sshd</code>（<code>ops</code> 讀不到 journal）。這次的清單只有少數幾項需要 root；TOML 那一篇的完整清單在專用帳號（加了 <code>systemd-journal</code> 與 <code>adm</code> 兩個群組的非 root 帳號）下，從 37 項 <code>PASS</code> 掉到 29 項。</li>
<li><strong>以 root 執行而不套 <code>systemd-run</code></strong>：這次沒有跑。結果依 TOML 那一篇的量測：檢查照常，dnf 的 log 與 sqlite 的 <code>-shm</code> 這類工具自己的寫入不再被擋下，檢查過程會在主機上留下寫入，只讀的保證沒有了。</li>
<li><strong>Ansible 集中執行，或每台用自己的 timer 執行</strong>：這次用的是 Ansible 集中執行。<a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈用 Ansible 對多台主機執行檢查腳本：整批的 exit code、每台的 rc 與執行身分的代價〉寫了集中執行的代價：控制端握有能 SSH 登入每台、再切成 root 的憑證。這次的 <code>ops</code> 帳號是 <code>NOPASSWD: ALL</code>，控制端的這把私鑰就等於兩台的 root。</li>
<li><strong>接上排程時讀結果的方式</strong>：<code>-o</code> 的一行輸出要用 <code>grep</code> 分出每台的 <code>rc</code>。設 <code>ANSIBLE_STDOUT_CALLBACK=json</code> 時，<code>ansible</code> 改印一份 JSON，每台的 <code>rc</code> 與 stdout 是其中的欄位，排程的腳本可以直接取值，代價是輸出不再適合人直接讀；改寫成 playbook，在任務上設 <code>failed_when: result.rc not in [0, 1]</code>（<code>result</code> 是 <code>register</code> 存下的結果），<code>WARN</code> 的 <code>rc=1</code> 就不算失敗，整批的 exit code 只在清單格式錯或腳本沒跑起來時變成 2，代價是 <code>WARN</code> 要另外從輸出裡讀（兩種都是推論，沒有實跑）。</li>
</ul>
<p>沙箱的屬性寫在 <code>check.sh</code> 裡有一個代價：同一組屬性在主機自己的 timer unit 裡還有一份，收緊其中一份時漏改另一份，沒有任何訊號。替代做法是把屬性放在主機上的一支包裝腳本或一個靜態的 service unit 裡，<code>check.sh</code> 與 timer 都呼叫它，屬性只有一份（推論，沒有實跑）。</p>
<h2 id="一台變成-warn一台清單格式錯一台連不上ansible-的-exit-code-與每台的-rc">一台變成 WARN、一台清單格式錯、一台連不上：ansible 的 exit code 與每台的 rc</h2>
<p>這一節依序製造門檻造成的 <code>WARN</code>、清單格式錯（被 <code>validate</code> 擋下的與在主機上手改的）、連不上的主機，每一種情形看 <code>ansible</code> 指令本身的 exit code，以及每台的標記與 <code>rc</code>。</p>
<p><strong>門檻讓 <code>mh-web1</code> 變成 <code>WARN</code></strong>。把 <code>host_vars/mh-web1.yml</code> 的 <code>min_free_bytes</code> 改成 1 TiB（<code>1099511627776</code>），行尾的註解 <code># 這台的根檔案系統至少留 2 GiB</code> 一起改成 1 TiB，之後讀這個檔的人才不會被舊的註解誤導。部署之前先只對 template 預覽：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 控制端容器裡</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible all -i inventory.yml -b -m template -a <span class="s2">&#34;src=host.toml.j2 dest=/etc/host-check/host.toml&#34;</span> --check --diff</span></span></code></pre></div><p><code>mh-both1</code> 印 <code>SUCCESS</code>、<code>&quot;changed&quot;: false</code>；<code>mh-web1</code> 印 <code>--- before: /etc/host-check/host.toml</code> 與 <code>+++ after:</code> 兩邊的差異，只有一行從 <code>-min_free_bytes = 2147483648</code> 換成 <code>+min_free_bytes = 1099511627776</code>，標成 <code>CHANGED</code>。這條指令的 exit code 是 0，有主機標成 <code>CHANGED</code> 時也是 0，所以拿它定期比對主機上的清單時，要讀輸出裡有沒有 <code>CHANGED</code>，只看 exit code 看不出哪一台的清單跟範本不同；判斷依據在 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈產生時驗證清單與比對主機上的清單〉。以 <code>ansible-playbook -i inventory.yml deploy.yml</code> 部署之後，再跑 <code>./check.sh</code>：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="ln">1</span><span class="cl">mh-both1 | CHANGED | rc=0 | (stdout) …summary: 11 PASS, 0 WARN, 3 MANUAL…
</span></span><span class="line"><span class="ln">2</span><span class="cl">mh-web1 | FAILED! =&gt; {&#34;changed&#34;: true, … &#34;rc&#34;: 1, … &#34;msg&#34;: &#34;non-zero return code&#34;, …}</span></span></code></pre></div><p><code>ansible</code> 的 exit code 是 2。<code>mh-web1</code> 的報告裡是 <code>WARN   filesystem/root  / 剩 77.7 GiB／共 117.6 GiB（下限 1024.0 GiB）</code>、<code>CHANGED filesystem/root  PASS→WARN</code> 與 <code>summary: 9 PASS, 1 WARN, 2 MANUAL</code>。<code>-o</code> 對成功的主機印一行 <code>CHANGED | rc=0 | (stdout) …</code>，對失敗的主機把整份結果印成一行 JSON，<code>rc</code> 是 JSON 裡的一個欄位，所以用 <code>grep</code> 分出每台的 <code>rc</code> 時，兩種形式都要比對。</p>
<p><strong>db 角色的 <code>exposure</code> 寫成 <code>private</code>，被 <code>validate</code> 擋下</strong>。把 <code>group_vars/db.yml</code> 的 <code>admin-only</code> 改成 <code>private</code> 再跑 <code>ansible-playbook -i inventory.yml deploy.yml</code>：exit code 2，<code>mh-both1</code> 在 template 那一步失敗，<code>&quot;msg&quot;: &quot;failed to validate&quot;</code>，stderr 的最後一行是 <code>hc.ManifestError: entrypoint[postgres]: exposure 要是 ['public', 'cdn-only', 'admin-only', 'loopback'] 之一，實際是 'private'</code>；<code>mh-web1</code> 不在 db 群組，<code>failed=0</code>。<code>mh-both1</code> 上的 <code>/etc/host-check/host.toml</code> 仍是上一版，三個 <code>exposure</code> 都沒有 <code>private</code>。把 <code>group_vars/db.yml</code> 改回 <code>admin-only</code>；主機上的清單沒有被改到，這裡不必重新部署。</p>
<p><strong>在主機上手改清單，檢查腳本以 exit code 2 結束</strong>。<code>validate</code> 擋下的是 Ansible 寫出的那一份；有人在主機上直接改檔，或另一條部署路徑沒有 <code>validate</code> 時，壞掉的清單照樣在主機上。用 <code>sed</code> 把 <code>mh-both1</code> 清單裡的 <code>exposure = &quot;admin-only&quot;</code> 改成 <code>exposure = &quot;private&quot;</code>，ssh 與 postgres 兩個入口都改到：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 開發機上執行：在 mh-both1 上直接改清單</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker <span class="nb">exec</span> mh-both1 sed -i <span class="s1">&#39;s/exposure = &#34;admin-only&#34;/exposure = &#34;private&#34;/&#39;</span> /etc/host-check/host.toml</span></span></code></pre></div><p><code>mh-web1</code> 維持 1 TiB 的門檻，再跑 <code>./check.sh</code>：</p>
<table>
  <thead>
      <tr>
          <th>主機</th>
          <th><code>-o</code> 的標記</th>
          <th><code>rc</code></th>
          <th>報告</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>mh-both1</code></td>
          <td><code>FAILED!</code></td>
          <td>2</td>
          <td>stderr 兩行 <code>manifest error: entrypoint[…]: exposure 要是 … 之一，實際是 'private'</code>，最後一行 <code>manifest error: 2 項，沒有執行任何檢查</code></td>
      </tr>
      <tr>
          <td><code>mh-web1</code></td>
          <td><code>FAILED!</code></td>
          <td>1</td>
          <td><code>summary: 9 PASS, 1 WARN, 2 MANUAL</code></td>
      </tr>
  </tbody>
</table>
<p><code>ansible</code> 的 exit code 是 2。兩台都標成 <code>FAILED!</code>，<code>rc</code> 分別是 2 與 1：<code>mh-both1</code> 在修好清單之前一項檢查都沒有跑，<code>mh-web1</code> 跑完了檢查、有一項 <code>WARN</code>（這次是門檻寫錯）。整批的 exit code 與每台的 <code>rc</code> 各自用來決定什麼，判斷依據在概念文章〈用 Ansible 對多台主機執行檢查腳本：整批的 exit code、每台的 rc 與執行身分的代價〉。接著依序做四件事，把兩台帶回全部 <code>PASS</code>：</p>
<ol>
<li><strong>比對主機上的清單</strong>：用門檻那一段只對 template 跑的 <code>--check --diff</code> 指令。<code>mh-both1</code> 標成 <code>CHANGED</code>，差異是兩行 <code>-exposure = &quot;private&quot;</code> 換成 <code>+exposure = &quot;admin-only&quot;</code>；<code>mh-web1</code> 是 <code>SUCCESS</code>，範本與主機上的門檻都是 1 TiB。exit code 0。</li>
<li><strong>門檻改回 2 GiB</strong>：<code>host_vars/mh-web1.yml</code> 的 <code>min_free_bytes</code> 與註解改回 <code>2147483648</code> 與 2 GiB。</li>
<li><strong>重新部署</strong>：<code>ansible-playbook -i inventory.yml deploy.yml</code>。這一步同時照範本把 <code>mh-both1</code> 的清單寫回 <code>admin-only</code>。不重新部署、只在主機上改回時，替代做法是在開發機上跑反向的 <code>sed</code>：<code>docker exec mh-both1 sed -i 's/exposure = &quot;private&quot;/exposure = &quot;admin-only&quot;/' /etc/host-check/host.toml</code>；已經重新部署的話不必跑。</li>
<li><strong>再跑 <code>./check.sh</code></strong>：exit code 回到 0，<code>mh-web1</code> 的報告多一行 <code>CHANGED filesystem/root  WARN→PASS</code>。</li>
</ol>
<p><strong><code>mh-web1</code> 連不上</strong>。在開發機上 <code>docker stop mh-web1</code> 之後跑 <code>./check.sh</code>：exit code 4，花 0.6 秒；<code>mh-web1</code> 印 <code>UNREACHABLE!: Failed to connect to the host via ssh: ssh: Could not resolve hostname mh-web1: Name or service not known</code>，沒有 <code>rc</code>；<code>mh-both1</code> 照常 <code>CHANGED | rc=0</code>。停掉的容器從 Docker network 的 DNS 消失，所以錯誤是解析不到主機名；真主機關機或網路斷了時，錯誤多半是連線逾時或被拒，exit code 同樣是 4（依 ansible-core 2.17.7 的原始碼 <code>lib/ansible/executor/task_queue_manager.py</code> 的 <code>RUN_*</code> 常數與這次的結果推論，真主機的訊息沒有實測）。同一次執行裡一台失敗、一台連不上時，依 ansible-core 2.17.7 的原始碼（<code>plugins/strategy/__init__.py</code>），有主機連不上時優先回 4，沒有才回 2；這次沒有實跑。照本文做不會同時出現兩種情形；排程裡同時有連不上與 <code>FAILED!</code> 時才用得到。</p>
<p>在開發機上 <code>docker start mh-web1</code>，等 systemd 啟動完成、sshd 開始接聽之後（<code>docker exec mh-web1 systemctl is-system-running --wait</code>），再跑一次 <code>./check.sh</code>：exit code 2，<code>mh-web1</code> 的 <code>rc</code> 是 1，報告是 <code>WARN   entrypoint/http:80/tcp  沒有程序在聽這個連接埠，exposure=cdn-only</code> 與 <code>CHANGED entrypoint/http:80/tcp  PASS→WARN</code>：代替 nginx 的 <code>stand-in-http</code> 是 <code>systemd-run</code> 建的暫時的 unit，容器重啟之後沒有被拉起來。這跟真主機重開機之後某個服務沒有設成開機啟動是同一種結果，清單裡寫了那個入口，它就被報出來。在開發機上重新起代替的接聽程式之後，<code>./check.sh</code> 回到 exit code 0：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 開發機上執行</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker <span class="nb">exec</span> mh-web1 systemd-run --unit<span class="o">=</span>stand-in-http python3 -m http.server <span class="m">80</span></span></span></code></pre></div><p>這次各情形的整批 exit code 與每台的結果：</p>
<table>
  <thead>
      <tr>
          <th>情形</th>
          <th><code>ansible</code> 的 exit code（整批）</th>
          <th>每台的標記與 <code>rc</code>（每台）</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>全部 <code>PASS</code> 與 <code>MANUAL</code></td>
          <td>0</td>
          <td>兩台 <code>CHANGED</code>、<code>rc=0</code></td>
      </tr>
      <tr>
          <td>一台的門檻造成 <code>WARN</code></td>
          <td>2</td>
          <td>該台 <code>FAILED!</code>、<code>rc</code> 1；另一台 <code>rc=0</code></td>
      </tr>
      <tr>
          <td>一台清單格式錯、另一台有 <code>WARN</code></td>
          <td>2</td>
          <td>兩台都 <code>FAILED!</code>，<code>rc</code> 分別是 2 與 1</td>
      </tr>
      <tr>
          <td>一台連不上</td>
          <td>4</td>
          <td>該台 <code>UNREACHABLE!</code>、沒有 <code>rc</code>；另一台 <code>rc=0</code></td>
      </tr>
      <tr>
          <td>不加 <code>-b</code>，<code>systemd-run</code> 起不來</td>
          <td>2（只對 <code>mh-web1</code> 跑）</td>
          <td>該台 <code>FAILED!</code>、<code>rc</code> 1，輸出沒有 <code>summary:</code></td>
      </tr>
      <tr>
          <td><code>--check --diff</code> 比對出清單被改過</td>
          <td>0</td>
          <td>該台 <code>CHANGED</code>，輸出有 <code>--- before</code> 與 <code>+++ after</code></td>
      </tr>
  </tbody>
</table>
<h2 id="容器跟真主機不同的地方以及在真主機上要另外確認的項目">容器跟真主機不同的地方，以及在真主機上要另外確認的項目</h2>
<p>照著在容器上做完不需要這一節；要把同樣的部署搬到真主機時，逐項確認。這次的受管主機是容器，有幾項結果在真主機上會不同，照著做在真主機上時，下面每一項要另外確認一次：</p>
<ul>
<li><strong>沙箱有沒有套上</strong>：這次的容器 <code>systemd-detect-virt</code> 印 <code>docker</code>，<code>ProtectSystem=strict</code> 底下 <code>touch /var/x</code> 回 <code>Read-only file system</code>。在 OrbStack 的 Linux machine 這類 LXC 環境裡同一條 <code>touch</code> 會成功、而且不報錯（<a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉記錄的 OrbStack Linux machine 實測），所以真主機上照樣先跑 <code>systemd-detect-virt</code> 與 <code>touch /var/x</code> 那兩條，再執行檢查。</li>
<li><strong>SELinux</strong>：容器裡沒有 SELinux（<a href="/blog/linux/dotfile/knowledge-cards/mandatory-access-control/" data-link-title="Mandatory Access Control（強制存取控制）：SELinux 與 AppArmor" data-link-desc="檔案權限看起來正確、程式以 root 執行，存取仍然被拒絕；從備份還原或搬移檔案之後服務回 403；或在 log 裡讀到 avc: denied、apparmor=&#34;DENIED&#34; 時">Mandatory Access Control（強制存取控制）：SELinux 與 AppArmor</a>），<code>getenforce</code> 這類檢查與 SELinux 對 <code>systemd-run</code> 執行的程序的限制都沒有量到。Oracle Linux 與 RHEL 的真主機預設是 enforcing（依 RHEL 文件〈Using SELinux〉，這次沒有在真主機上確認），部署到 <code>/opt/host-check</code> 的腳本由 <code>systemd-run</code> 執行時，SELinux 是否拒絕它的存取，要在真主機上跑一次檢查，再用 <code>ausearch -m avc -ts recent</code> 查有沒有拒絕的紀錄。</li>
<li><strong><code>log/sshd</code> 被 Ansible 自己的紀錄滿足</strong>：容器遮蔽了 <code>systemd-logind</code>，SSH 登入之後的程序留在 <code>sshd.service</code> 的 cgroup 裡，Ansible 模組寫進 journal 的紀錄因此算在 <code>sshd.service</code> 底下。真主機上 logind 會替每個登入建一個 <code>session-&lt;編號&gt;.scope</code>，那些紀錄落在 session 底下（從 logind 的機制推的，沒有在真主機上比對）；在真主機上用 <code>journalctl -u sshd -n 5</code> 看最新幾筆是 sshd 自己的紀錄還是 Ansible 的，確認這一項量的是 sshd。</li>
<li><strong>雲端防火牆</strong>：清單裡每個入口的 <code>cloud-firewall</code> 那一層都是 <code>MANUAL</code>，這次沒有任何雲端資源可以比對。主機與規則由 Terraform 建立時，<code>terraform plan</code> 比對得到與比對不到的範圍在 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈清單裡要到主控台確認的項目與 terraform plan 的比對範圍〉。</li>
<li><strong>主機防火牆</strong>：容器裡沒有 firewalld，清單也沒有寫主機防火牆那一層；真主機的清單在 <code>enforced_by</code> 加上 <code>firewalld</code>，並在那一層的 <code>[[entrypoint.verify]]</code> 寫 <code>where = &quot;host&quot;</code> 與探針（<code>command</code> 欄位寫要執行的 <code>firewall-cmd --query-*</code> 指令，<code>expect_exit</code> 這類欄位寫什麼結果算通過）。</li>
<li><strong>journal 只在記憶體裡</strong>：這次的容器與 Oracle Cloud 的 Oracle Linux 10.2 映像檔預設都只把 journal 存在記憶體（<a href="/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算</a>〈journald 的紀錄存在記憶體還是磁碟〉），<code>/var/log/journal</code> 不存在，<code>accumulation/journal</code> 以 <code>missing_ok</code> 判成 <code>PASS</code>。開了 <code>Storage=persistent</code>、或發行版預設建了 <code>/var/log/journal</code> 的主機（Debian 系；同一節寫了 Debian 12、13 與 Ubuntu 24.04 的 systemd 套件在安裝時建立這個目錄，以及在 Oracle Cloud 的 Ubuntu 24.04 映像檔上的確認）上，這一項才量得到實際大小，1 GiB 的上限要照那台主機的磁碟與 <code>SystemMaxUse=</code> 重訂。</li>
<li><strong>連線位址與延遲</strong>：<code>ansible_host</code> 這次是容器名，部署 1.9 秒、檢查 0.9 秒都沒有經過真正的網路；真主機上 <code>ansible_host</code> 填 IP 或 DNS 名稱，時間會加上每台的 SSH 連線。</li>
<li><strong>sudo 與 host key</strong>：這次的 <code>ops</code> 是 <code>NOPASSWD: ALL</code>，<code>known_hosts</code> 由 <code>ssh-keyscan</code> 直接寫入，兩者在真主機上都要重新決定；host key 的做法在〈這一節的其他做法：自己的金鑰與寫入 known_hosts 的方式〉，sudo 要密碼時的做法在〈這一節的其他做法：容器或 VM、作業系統、遮蔽的 unit、privileged、sudo 密碼、安裝版本與雜湊、金鑰密碼〉。</li>
</ul>
<h2 id="清理terraform-destroy容器network映像檔與金鑰">清理：terraform destroy、容器、network、映像檔與金鑰</h2>
<p>清理的順序是先 <code>terraform destroy</code>、再刪容器：destroy 要在控制端容器裡執行，控制端先刪掉的話 Terraform 不在了，State 留在工作目錄裡沒有人處理。這次 State 只記著一份本機檔案；真環境的 State 記著雲端主機，先 destroy 才不會留下還在計費的資源。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 開發機上</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">docker <span class="nb">exec</span> -w /work/tf mh-ctl terraform destroy -auto-approve   <span class="c1"># 印 Destroy complete! Resources: 1 destroyed.</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl">docker rm -f mh-ctl mh-web1 mh-both1
</span></span><span class="line"><span class="ln">4</span><span class="cl">docker network rm mh-net
</span></span><span class="line"><span class="ln">5</span><span class="cl">docker rmi mh-host:ol10 mh-ctl:1
</span></span><span class="line"><span class="ln">6</span><span class="cl">docker ps -a --filter <span class="nv">name</span><span class="o">=</span>mh- -q<span class="p">;</span> docker network ls --filter <span class="nv">name</span><span class="o">=</span>mh- -q<span class="p">;</span> docker images <span class="s1">&#39;mh-*&#39;</span> -q<span class="p">;</span> docker volume ls --filter <span class="nv">name</span><span class="o">=</span>mh- -q
</span></span><span class="line"><span class="ln">7</span><span class="cl">rm -rf &lt;工作目錄&gt; &lt;工作目錄的上一層&gt;/host-image &lt;工作目錄的上一層&gt;/ctl-image &lt;暫存目錄&gt;/keys</span></span></code></pre></div><p><code>terraform destroy</code> 刪掉的是 <code>local_file.inventory</code>，之後在開發機上 <code>ls &lt;工作目錄&gt;/ansible/inventory.yml</code> 回 <code>No such file or directory</code>。<code>docker ps -a --filter name=mh- -q</code> 那一行的四條查詢都沒有輸出，代表這次建的容器、network、映像檔與 volume 都清掉了（這次的容器沒有掛 named volume，<code>docker volume ls</code> 那一條是確認沒有遺漏）。<code>docker build</code> 留下的建置快取沒有清：清快取要用 <code>docker builder prune</code>，它不分是哪一次建置留下的，會一起清掉開發機上其他專案的快取，所以這次留著。底層映像檔 <code>oraclelinux:10</code> 與 <code>python:3.12-slim</code> 也留著：開發機上可能有別的專案在用它們，要刪時執行 <code>docker rmi oraclelinux:10 python:3.12-slim</code>。工作目錄裡有 Terraform 的 State 與產生的清單、暫存目錄裡有私鑰，<code>host-image/</code> 與 <code>ctl-image/</code> 是兩個映像檔的建置目錄，<code>rm -rf</code> 那一行把它們一起刪掉；<code>&lt;工作目錄&gt;</code>、<code>&lt;工作目錄的上一層&gt;</code> 與 <code>&lt;暫存目錄&gt;</code> 換成這次實際用的路徑，刪之前先 <code>ls</code> 一次確認是這次建的目錄。</p>
<h2 id="延伸閱讀分層與合併的判斷依據清單的欄位與檢查腳本terraform-與-ansible-的卡">延伸閱讀：分層與合併的判斷依據、清單的欄位與檢查腳本、Terraform 與 Ansible 的卡</h2>
<ul>
<li><a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>：每台相同、依角色不同、每台不同的分類、合併放在部署時的理由、<code>to_json</code> 的跳脫與 YAML 的隱式型別、exit code 2 與 4 的讀法與集中執行的安全代價</li>
<li><a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>：清單每一段的欄位、檢查腳本的白名單、<code>systemd-run</code> 各屬性在 Oracle Linux 10 容器上的實測效果與結果檔</li>
<li><a href="/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</a>：這篇部署與執行的那支腳本的完整原始碼，<code>load_manifest</code> 的逐欄驗證、<code>summary:</code> 與各個 exit code 的出口、結果檔的寫法，以及 <code>runpy</code> 載入時不執行主程式的條件</li>
<li><a href="/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">Configuration Management（組態管理）</a>：inventory、群組變數與主機變數各放什麼；同一個變數在幾層都有值時的合併順序，在概念文章〈用 Ansible 的 inventory、群組變數與範本產生每台的清單〉</li>
<li><a href="/blog/devops/infra/knowledge-cards/state/" data-link-title="State（IaC 狀態檔）" data-link-desc="IaC 工具用來記錄每個納管資源在雲端真實樣貌的快照，是比對差異與排定操作順序的依據">State（IaC 狀態檔）</a>：IaC 工具記錄每個納管資源上一次 apply 之後樣貌的快照，以及它的存放位置、加密與鎖機制</li>
<li><a href="/blog/devops/infra/knowledge-cards/terraform-plan-apply/" data-link-title="terraform plan / apply" data-link-desc="IaC 的兩個核心操作：plan 只看不動（產出差異報告）、apply 真的動（執行差異）">terraform plan / apply</a>：plan 比對 State、雲端現況與設定檔產出差異報告而不改動任何資源，apply 執行那份差異；手改的 inventory 被 plan 判成重新建立的機制，在概念文章〈主機由 Terraform 建立時由它的輸出產生 inventory〉</li>
</ul>
]]></content:encoded></item><item><title>只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</title><link>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/</link><pubDate>Mon, 26 Oct 2026 00:00:00 +0800</pubDate><guid>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/</guid><description>&lt;blockquote>
&lt;p>&lt;strong>事件說明&lt;/strong>：2026-10-26，把作者私有專案裡的檢查腳本 &lt;code>host-check.py&lt;/code> 在這一天的版本（commit &lt;code>3fc41c0&lt;/code>）整理成這篇。這一版的單元測試在一台 Apple Silicon 的 macOS 開發機上，用 uv 分別起 Python 3.12 與 Python 3.15 各執行一次；這一版在 Oracle Linux 10 的容器裡執行過；一台 Oracle Linux 10.2 的雲端主機上執行過的，是修正「權限擋下的目錄被判成路徑不存在」之前的版本。&lt;/p>&lt;/blockquote>
&lt;p>這篇是主機維護清單的只讀檢查腳本 &lt;code>host-check.py&lt;/code> 的完整原始碼，依腳本自己的結構分節。每一個 &lt;code>##&lt;/code> 節先寫這一段程式負責什麼、它滿足的是清單或執行環境的哪一項需求，接著是這一段的原始碼，之後說明從程式碼不容易看出來的地方：為什麼這樣判斷、換成另一種寫法會讓哪一種寫入或誤判通過、哪一項單元測試鎖住這個行為。各節依序是：檔頭的說明與共用常數；照 &lt;code>getopt_long&lt;/code> 語法切分選項的函式；&lt;code>systemctl&lt;/code>、&lt;code>journalctl&lt;/code>、&lt;code>firewall-cmd&lt;/code>、&lt;code>ss&lt;/code>、&lt;code>find&lt;/code>、&lt;code>sqlite3&lt;/code>、&lt;code>openssl&lt;/code>、&lt;code>sshd&lt;/code>、&lt;code>needs-restarting&lt;/code>、&lt;code>dnf&lt;/code>、&lt;code>rpm&lt;/code>、&lt;code>ausearch&lt;/code> 各自的選項規則，以及不另外比對選項的工具；判定一條指令是否只讀的 &lt;code>readonly_violation&lt;/code> 與腳本自己組出來的指令；清單的載入與驗證；執行指令的 &lt;code>Runner&lt;/code> 與時鐘；&lt;code>Checker&lt;/code> 對入口與接聽中的連接埠、log、會累積的資料、檔案系統、自動更新、暫停與恢復步驟的檢查；路徑的展開與大小的量測；時間字串的解析與組輸出用的小函式；結果檔的讀寫與跟前一次的差異；命令列與 exit code；最後是這一版的已知限制與重現方式，依清單的驗證、指令的執行與 exit code、各項檢查、結果檔分組。&lt;/p>
&lt;p>各節的 &lt;code>python&lt;/code> 程式碼區塊照順序接起來就是完整的腳本。區塊內的空行照原檔保留；兩個區塊之間夾著說明的位置，原檔是空行，接的時候補回去：下一個區塊以縮排開頭（&lt;code>Checker&lt;/code> 類別裡的方法）時補一個空行，其餘補兩個空行。區塊之間補幾個空行只影響跟原檔逐字比對的結果，不影響執行。照這條規則接出來的檔案是 1442 行，sha256 是 &lt;code>22228d77ee7bc0d126401d04fc3a73b37e0a5768566abefa2b2b9dce69a0761d&lt;/code>；這是本篇版本（拿掉 &lt;code># i18n-exempt&lt;/code> 標記、換掉兩處票號之後）的值，不是私有專案裡原檔的值。跟原檔相比改了兩處：行尾給私有專案的翻譯檢查工具用的 &lt;code># i18n-exempt&lt;/code> 標記拿掉了；檔頭說明裡兩處專案內部的工作票號，換成不帶票號的寫法。腳本只用 Python 標準函式庫，需要 Python 3.11 以上（解析 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/toml/" data-link-title="TOML" data-link-desc="讀到 .toml 設定檔、要判斷一個值會被解析成什麼型別、要在 JSON、YAML、TOML 之間選設定檔格式，或解析器版本對不上而報錯時">TOML&lt;/a> 的 &lt;code>tomllib&lt;/code> 在 3.11 加入）。單元測試的檔案沒有收進這篇，說明裡提到的測試以測試函式的名字指稱，並寫出它斷言的行為；每一個 &lt;code>##&lt;/code> 節第一次提到時寫成「單元測試 &lt;code>test_…&lt;/code>（本篇沒有收錄）」。74 項單元測試在 Python 3.12 與 3.15 都通過；以 root 執行時，&lt;code>test_glob_in_unlistable_dir_is_unreadable&lt;/code>、&lt;code>test_unreadable_glob_with_missing_ok_does_not_pass&lt;/code>、&lt;code>test_literal_path_under_unreadable_parent_is_unreadable&lt;/code> 這三項會失敗：它們靠目錄權限擋住讀取，而 root 不受目錄權限限制，測試裡的前提檢查寫明以 root 執行時這幾項失敗、不跳過（讀碼）。&lt;/p>
&lt;p>文中的行為標兩種來源。「實跑」是實際執行過的結果：沒有另外註明環境的，是在一台 Apple Silicon 的 macOS 26.5 開發機上，以 Python 3.15.0 執行這一版的腳本，把它載入成模組、直接呼叫函式或方法，或從命令列跑整支腳本（Python 的版本差異另外用 uv 起 3.10.18、3.11.13、3.12.11）；在別的環境跑的，就地寫出環境。「讀碼」是從程式碼推得、沒有執行過的行為。&lt;/p>
&lt;p>清單的格式與欄位、白名單要擋下哪些寫法與理由、輸出與 exit code 的讀法、結果檔，以及 systemd 沙箱各屬性實跑的效果，寫在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本&lt;/a>，這篇不重寫那些理由，各節連到對應的段落。把這支腳本與每台的清單部署到多台主機、以 sudo 與 &lt;code>systemd-run&lt;/code> 執行它的步驟與輸出，在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理&lt;/a>。&lt;/p>
&lt;p>接起來的腳本要一份清單才跑得起來：它預設讀腳本所在目錄的 &lt;code>host.toml&lt;/code>，沒有清單或清單格式錯時以 exit 2 結束、一項檢查都不執行。最小的清單從 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理&lt;/a>〈變數檔與範本：檢查腳本實際要求的欄位、兼任兩種角色的主機〉的 &lt;code>host.toml.j2&lt;/code> 取得：把裡面的 &lt;code>{{ }}&lt;/code> 換成這台主機實際的值，就是一份這支腳本載入得過的清單；哪些段落必填、哪些可以不寫，在本篇〈清單的載入與驗證：Probe、_Validator、load_manifest 與 validate〉。&lt;/p></description><content:encoded><![CDATA[<blockquote>
<p><strong>事件說明</strong>：2026-10-26，把作者私有專案裡的檢查腳本 <code>host-check.py</code> 在這一天的版本（commit <code>3fc41c0</code>）整理成這篇。這一版的單元測試在一台 Apple Silicon 的 macOS 開發機上，用 uv 分別起 Python 3.12 與 Python 3.15 各執行一次；這一版在 Oracle Linux 10 的容器裡執行過；一台 Oracle Linux 10.2 的雲端主機上執行過的，是修正「權限擋下的目錄被判成路徑不存在」之前的版本。</p></blockquote>
<p>這篇是主機維護清單的只讀檢查腳本 <code>host-check.py</code> 的完整原始碼，依腳本自己的結構分節。每一個 <code>##</code> 節先寫這一段程式負責什麼、它滿足的是清單或執行環境的哪一項需求，接著是這一段的原始碼，之後說明從程式碼不容易看出來的地方：為什麼這樣判斷、換成另一種寫法會讓哪一種寫入或誤判通過、哪一項單元測試鎖住這個行為。各節依序是：檔頭的說明與共用常數；照 <code>getopt_long</code> 語法切分選項的函式；<code>systemctl</code>、<code>journalctl</code>、<code>firewall-cmd</code>、<code>ss</code>、<code>find</code>、<code>sqlite3</code>、<code>openssl</code>、<code>sshd</code>、<code>needs-restarting</code>、<code>dnf</code>、<code>rpm</code>、<code>ausearch</code> 各自的選項規則，以及不另外比對選項的工具；判定一條指令是否只讀的 <code>readonly_violation</code> 與腳本自己組出來的指令；清單的載入與驗證；執行指令的 <code>Runner</code> 與時鐘；<code>Checker</code> 對入口與接聽中的連接埠、log、會累積的資料、檔案系統、自動更新、暫停與恢復步驟的檢查；路徑的展開與大小的量測；時間字串的解析與組輸出用的小函式；結果檔的讀寫與跟前一次的差異；命令列與 exit code；最後是這一版的已知限制與重現方式，依清單的驗證、指令的執行與 exit code、各項檢查、結果檔分組。</p>
<p>各節的 <code>python</code> 程式碼區塊照順序接起來就是完整的腳本。區塊內的空行照原檔保留；兩個區塊之間夾著說明的位置，原檔是空行，接的時候補回去：下一個區塊以縮排開頭（<code>Checker</code> 類別裡的方法）時補一個空行，其餘補兩個空行。區塊之間補幾個空行只影響跟原檔逐字比對的結果，不影響執行。照這條規則接出來的檔案是 1442 行，sha256 是 <code>22228d77ee7bc0d126401d04fc3a73b37e0a5768566abefa2b2b9dce69a0761d</code>；這是本篇版本（拿掉 <code># i18n-exempt</code> 標記、換掉兩處票號之後）的值，不是私有專案裡原檔的值。跟原檔相比改了兩處：行尾給私有專案的翻譯檢查工具用的 <code># i18n-exempt</code> 標記拿掉了；檔頭說明裡兩處專案內部的工作票號，換成不帶票號的寫法。腳本只用 Python 標準函式庫，需要 Python 3.11 以上（解析 <a href="/blog/devops/infra/knowledge-cards/toml/" data-link-title="TOML" data-link-desc="讀到 .toml 設定檔、要判斷一個值會被解析成什麼型別、要在 JSON、YAML、TOML 之間選設定檔格式，或解析器版本對不上而報錯時">TOML</a> 的 <code>tomllib</code> 在 3.11 加入）。單元測試的檔案沒有收進這篇，說明裡提到的測試以測試函式的名字指稱，並寫出它斷言的行為；每一個 <code>##</code> 節第一次提到時寫成「單元測試 <code>test_…</code>（本篇沒有收錄）」。74 項單元測試在 Python 3.12 與 3.15 都通過；以 root 執行時，<code>test_glob_in_unlistable_dir_is_unreadable</code>、<code>test_unreadable_glob_with_missing_ok_does_not_pass</code>、<code>test_literal_path_under_unreadable_parent_is_unreadable</code> 這三項會失敗：它們靠目錄權限擋住讀取，而 root 不受目錄權限限制，測試裡的前提檢查寫明以 root 執行時這幾項失敗、不跳過（讀碼）。</p>
<p>文中的行為標兩種來源。「實跑」是實際執行過的結果：沒有另外註明環境的，是在一台 Apple Silicon 的 macOS 26.5 開發機上，以 Python 3.15.0 執行這一版的腳本，把它載入成模組、直接呼叫函式或方法，或從命令列跑整支腳本（Python 的版本差異另外用 uv 起 3.10.18、3.11.13、3.12.11）；在別的環境跑的，就地寫出環境。「讀碼」是從程式碼推得、沒有執行過的行為。</p>
<p>清單的格式與欄位、白名單要擋下哪些寫法與理由、輸出與 exit code 的讀法、結果檔，以及 systemd 沙箱各屬性實跑的效果，寫在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>，這篇不重寫那些理由，各節連到對應的段落。把這支腳本與每台的清單部署到多台主機、以 sudo 與 <code>systemd-run</code> 執行它的步驟與輸出，在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>。</p>
<p>接起來的腳本要一份清單才跑得起來：它預設讀腳本所在目錄的 <code>host.toml</code>，沒有清單或清單格式錯時以 exit 2 結束、一項檢查都不執行。最小的清單從 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈變數檔與範本：檢查腳本實際要求的欄位、兼任兩種角色的主機〉的 <code>host.toml.j2</code> 取得：把裡面的 <code>{{ }}</code> 換成這台主機實際的值，就是一份這支腳本載入得過的清單；哪些段落必填、哪些可以不寫，在本篇〈清單的載入與驗證：Probe、_Validator、load_manifest 與 validate〉。</p>
<h2 id="檔頭的說明匯入的模組與共用常數">檔頭的說明、匯入的模組與共用常數</h2>
<p>檔頭的 docstring 是寫給維護的人讀的使用說明：用法、輸出的格式、結果檔、systemd 沙箱要加的屬性、exit code 與只讀的保證；這些行為的理由與實跑結果在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。docstring 之後是匯入的模組與整支腳本共用的常數。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="ch">#!/usr/bin/env python3</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="s2">&#34;&#34;&#34;web-demo 主機的只讀維護檢查。
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="s2">在主機上讀同目錄的 host.toml，逐項執行 where = &#34;host&#34; 的只讀檢查，主機外
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="s2">（OCI Console、Cloudflare dashboard、開發機）的項目印成 MANUAL。
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="s2">用法（在主機上，以 root 執行；firewalld 查詢、sshd -T、collector 的資料庫、nginx 的 log 目錄、
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="s2">audit.log 與 journal 一般帳號讀不到，專用帳號的實測見 README〈以專用帳號執行〉）：
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="s2">    sudo python3 host-check.py                       # 例行維護：入口、log、累積的資料、更新
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="s2">    sudo python3 host-check.py --phase suspend       # 暫停上線前要做的步驟
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="s2">    sudo python3 host-check.py --phase resume        # 恢復上線後要做的步驟
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="s2">    sudo python3 host-check.py --manifest other.toml --set repo_dir=/opt/monitor
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="s2">    sudo python3 host-check.py --state-file PATH     # 結果檔換路徑（預設 /var/lib/host-check/last.json）
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="s2">    sudo python3 host-check.py --no-state            # 不讀也不寫結果檔
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="s2">輸出：每項一行 `PASS` / `WARN` / `MANUAL`、項目名與理由；接著是與前一次的差（`since:` 開頭，
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="s2">其後每個差一行：`CHANGED` 狀態改變、`DELTA` 值改變、`NEW` 新出現、`GONE` 消失）；結尾一行
</span></span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="s2">`summary: N PASS, M WARN, K MANUAL`；沒有印出 summary 代表腳本中途中止。
</span></span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">21</span><span class="cl"><span class="s2">結果檔：每次執行結束把這次的計數與量到的值（探針值、累積的 bytes、可用 bytes）寫進結果檔
</span></span></span><span class="line"><span class="ln">22</span><span class="cl"><span class="s2">（JSON，每個 phase 一段，只替換本次的 phase），下一次拿它算差。目錄不存在時以 0700 建立，檔案
</span></span></span><span class="line"><span class="ln">23</span><span class="cl"><span class="s2">0600，用同目錄暫存檔加 os.replace 寫入。目錄已存在時不改它的權限：由 systemd 的 StateDirectory=
</span></span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="s2">建立的目錄，權限照 StateDirectoryMode=（預設 0755，每次啟動都會把既有目錄改回這個值），沙箱裡
</span></span></span><span class="line"><span class="ln">25</span><span class="cl"><span class="s2">執行時要設 StateDirectoryMode=0700。讀不到、壞掉或寫不進去只在 stderr 印 `note:`，不改變
</span></span></span><span class="line"><span class="ln">26</span><span class="cl"><span class="s2">exit code；清單本身只描述要檢查什麼，不存放結果。
</span></span></span><span class="line"><span class="ln">27</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">28</span><span class="cl"><span class="s2">systemd 沙箱（在 Oracle Linux 10 容器裡實測）：ProtectSystem=strict 下要加 ReadWritePaths=/var/log /var/cache/dnf
</span></span></span><span class="line"><span class="ln">29</span><span class="cl"><span class="s2">（dnf needs-restarting 要寫 dnf.log 與 metadata lock），StateDirectory=host-check 加 StateDirectoryMode=0700；
</span></span></span><span class="line"><span class="ln">30</span><span class="cl"><span class="s2">NoNewPrivileges、PrivateTmp、IPAddressDeny=any 不影響檢查；不開 PrivateNetwork（ss 看不到主機的 socket，
</span></span></span><span class="line"><span class="ln">31</span><span class="cl"><span class="s2">入口與 listen-audit 的檢查都會失準）。完整設定見 README〈在 systemd 沙箱裡執行〉。
</span></span></span><span class="line"><span class="ln">32</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">33</span><span class="cl"><span class="s2">exit code：
</span></span></span><span class="line"><span class="ln">34</span><span class="cl"><span class="s2">    0  全部 PASS，或只有 PASS 與 MANUAL
</span></span></span><span class="line"><span class="ln">35</span><span class="cl"><span class="s2">    1  至少一項 WARN
</span></span></span><span class="line"><span class="ln">36</span><span class="cl"><span class="s2">    2  清單格式錯（缺必填欄位、型別錯、要執行的指令不在只讀白名單裡），一項都沒有執行
</span></span></span><span class="line"><span class="ln">37</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">38</span><span class="cl"><span class="s2">只讀的保證：
</span></span></span><span class="line"><span class="ln">39</span><span class="cl"><span class="s2">    - 指令不經過 shell（shlex 切詞後直接 exec），沒有管線、轉向與命令替換。
</span></span></span><span class="line"><span class="ln">40</span><span class="cl"><span class="s2">    - 要執行的每一條指令在執行任何一條之前先比對白名單（READONLY_RULES）；
</span></span></span><span class="line"><span class="ln">41</span><span class="cl"><span class="s2">      不在白名單裡、或帶了會改變狀態的參數（find -delete、
</span></span></span><span class="line"><span class="ln">42</span><span class="cl"><span class="s2">      firewall-cmd --add-*、sqlite3 沒有同時帶 -init /dev/null、-readonly 與 -safe 這類）時，整份清單判成格式錯。
</span></span></span><span class="line"><span class="ln">43</span><span class="cl"><span class="s2">    - systemctl、dnf、rpm、journalctl、ss、find 的選項是列允許：依各工具自己的選項語法（getopt_long，
</span></span></span><span class="line"><span class="ln">44</span><span class="cl"><span class="s2">      吃值的選項會吃掉下一個字，值可黏在選項後）切分，表外的選項一律拒絕，所以工具日後新增的
</span></span></span><span class="line"><span class="ln">45</span><span class="cl"><span class="s2">      寫入選項預設就是拒絕（例：rpm --pipe/--eval/-D、systemctl -H/-M、dnf -x、journalctl --cursor-file）。
</span></span></span><span class="line"><span class="ln">46</span><span class="cl"><span class="s2">    - sshd 只接受單獨的 -T；指令的路徑只接受裸名或 /usr/bin、/usr/sbin、/bin、/sbin 下的檔案。
</span></span></span><span class="line"><span class="ln">47</span><span class="cl"><span class="s2">    - command 一律會被執行（且必須通過白名單）；manual_command、purge_command 是要人執行的動作，
</span></span></span><span class="line"><span class="ln">48</span><span class="cl"><span class="s2">      一律只印出來。只印的位置（[[log]]、[[suspend]] / [[resume]]、主機外的 [[entrypoint.verify]]）
</span></span></span><span class="line"><span class="ln">49</span><span class="cl"><span class="s2">      寫舊名 command 判成格式錯。
</span></span></span><span class="line"><span class="ln">50</span><span class="cl"><span class="s2">    - 寫結果檔之外不改動任何受檢狀態；結果檔是腳本自己唯一會寫的檔案。例外：collector 停止後
</span></span></span><span class="line"><span class="ln">51</span><span class="cl"><span class="s2">      資料庫的 -shm、-wal 已被收掉，此時 sqlite3 -readonly 會自己建出這兩個檔（擁有者同資料庫）；
</span></span></span><span class="line"><span class="ln">52</span><span class="cl"><span class="s2">      在唯讀檔案系統（ProtectSystem=strict）上建不出來，sqlite3 exit 14，該項印成 WARN。
</span></span></span><span class="line"><span class="ln">53</span><span class="cl"><span class="s2">    - 檔案大小與時間用 Python 的 os.lstat 量，不執行 du。
</span></span></span><span class="line"><span class="ln">54</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln">55</span><span class="cl"><span class="s2">只用標準函式庫，需要 Python 3.11 以上（tomllib）。
</span></span></span><span class="line"><span class="ln">56</span><span class="cl"><span class="s2">&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">57</span><span class="cl">
</span></span><span class="line"><span class="ln">58</span><span class="cl"><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">annotations</span>
</span></span><span class="line"><span class="ln">59</span><span class="cl">
</span></span><span class="line"><span class="ln">60</span><span class="cl"><span class="kn">import</span> <span class="nn">argparse</span>
</span></span><span class="line"><span class="ln">61</span><span class="cl"><span class="kn">import</span> <span class="nn">configparser</span>
</span></span><span class="line"><span class="ln">62</span><span class="cl"><span class="kn">import</span> <span class="nn">datetime</span> <span class="k">as</span> <span class="nn">dt</span>
</span></span><span class="line"><span class="ln">63</span><span class="cl"><span class="kn">import</span> <span class="nn">glob</span>
</span></span><span class="line"><span class="ln">64</span><span class="cl"><span class="kn">import</span> <span class="nn">json</span>
</span></span><span class="line"><span class="ln">65</span><span class="cl"><span class="kn">import</span> <span class="nn">os</span>
</span></span><span class="line"><span class="ln">66</span><span class="cl"><span class="kn">import</span> <span class="nn">re</span>
</span></span><span class="line"><span class="ln">67</span><span class="cl"><span class="kn">import</span> <span class="nn">shlex</span>
</span></span><span class="line"><span class="ln">68</span><span class="cl"><span class="kn">import</span> <span class="nn">stat</span>
</span></span><span class="line"><span class="ln">69</span><span class="cl"><span class="kn">import</span> <span class="nn">subprocess</span>
</span></span><span class="line"><span class="ln">70</span><span class="cl"><span class="kn">import</span> <span class="nn">sys</span>
</span></span><span class="line"><span class="ln">71</span><span class="cl"><span class="kn">import</span> <span class="nn">tempfile</span>
</span></span><span class="line"><span class="ln">72</span><span class="cl"><span class="kn">import</span> <span class="nn">time</span>
</span></span><span class="line"><span class="ln">73</span><span class="cl"><span class="kn">import</span> <span class="nn">tomllib</span>
</span></span><span class="line"><span class="ln">74</span><span class="cl"><span class="kn">from</span> <span class="nn">dataclasses</span> <span class="kn">import</span> <span class="n">dataclass</span><span class="p">,</span> <span class="n">field</span>
</span></span><span class="line"><span class="ln">75</span><span class="cl"><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">Any</span><span class="p">,</span> <span class="n">Callable</span>
</span></span><span class="line"><span class="ln">76</span><span class="cl">
</span></span><span class="line"><span class="ln">77</span><span class="cl"><span class="n">WHERE_VALUES</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;host&#34;</span><span class="p">,</span> <span class="s2">&#34;external&#34;</span><span class="p">,</span> <span class="s2">&#34;console&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">78</span><span class="cl"><span class="n">EXPOSURE_VALUES</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;public&#34;</span><span class="p">,</span> <span class="s2">&#34;cdn-only&#34;</span><span class="p">,</span> <span class="s2">&#34;admin-only&#34;</span><span class="p">,</span> <span class="s2">&#34;loopback&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">79</span><span class="cl"><span class="n">PROTOCOL_VALUES</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;tcp&#34;</span><span class="p">,</span> <span class="s2">&#34;udp&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">80</span><span class="cl"><span class="n">BILLED_VALUES</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;yes&#34;</span><span class="p">,</span> <span class="s2">&#34;no&#34;</span><span class="p">,</span> <span class="s2">&#34;unknown&#34;</span><span class="p">,</span> <span class="s2">&#34;n/a&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">81</span><span class="cl"><span class="n">MEASURE_VALUES</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;lines&#34;</span><span class="p">,</span> <span class="s2">&#34;value&#34;</span><span class="p">,</span> <span class="s2">&#34;age_days&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">82</span><span class="cl"><span class="n">PHASES</span> <span class="o">=</span> <span class="p">(</span><span class="s2">&#34;maintain&#34;</span><span class="p">,</span> <span class="s2">&#34;suspend&#34;</span><span class="p">,</span> <span class="s2">&#34;resume&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">83</span><span class="cl"><span class="n">COMMAND_TIMEOUT_S</span> <span class="o">=</span> <span class="mi">30</span>
</span></span><span class="line"><span class="ln">84</span><span class="cl"><span class="n">DEFAULT_STATE_FILE</span> <span class="o">=</span> <span class="s2">&#34;/var/lib/host-check/last.json&#34;</span>
</span></span><span class="line"><span class="ln">85</span><span class="cl"><span class="n">STATE_FORMAT</span> <span class="o">=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">86</span><span class="cl">
</span></span><span class="line"><span class="ln">87</span><span class="cl"><span class="n">VAR_RE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;\{\{([A-Za-z_][A-Za-z0-9_]*)\}\}&#34;</span><span class="p">)</span></span></span></code></pre></div><p>docstring 裡的 README〈以專用帳號執行〉與 README〈在 systemd 沙箱裡執行〉是私有專案的 README，沒有公開；改用專用帳號執行時哪些檢查讀得到、哪些讀不到，以及沙箱每一個屬性的實跑效果，寫在清單寫成 TOML 的那篇概念文章的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。docstring 裡的 <code>web-demo</code> 是示範清單所描述的那台主機上跑的服務，collector 是那台主機上寫 SQLite 資料庫的程式。</p>
<p>docstring 裡有幾句跟程式實際的行為不一致，程式碼照原檔保留，實際行為以後面的節為準：</p>
<ul>
<li><strong>「systemctl、dnf、rpm、journalctl、ss、find 的選項是列允許：依各工具自己的選項語法（getopt_long，……）切分」</strong>：「列允許」成立，<code>find</code> 也是列允許；而 <code>find</code> 不照 <code>getopt_long</code> 語法切分，它的規則自己逐個參數比對，寫在〈find 的運算式〉。照 <code>getopt_long</code> 語法切分的是 <code>systemctl</code>、<code>journalctl</code>、<code>ss</code>、<code>needs-restarting</code>、<code>dnf</code>、<code>rpm</code> 這六支，寫在〈選項的切分：OptionSpec 與 getopt_long 語法〉。</li>
<li><strong>exit code 2 的「清單格式錯（……），一項都沒有執行」</strong>：清單格式錯時成立。exit 2 另有一個出口：清單通過載入時的比對、執行時 <code>Runner</code> 的白名單又拒絕了一條指令，這時在它之前的指令已經執行過，寫在〈命令列與 exit code：main〉。</li>
<li><strong>exit code 1 的「至少一項 WARN」</strong>：檢查中途丟出沒有接住的例外時，Python 也以 exit 1 結束，這時 stdout 沒有 <code>summary:</code> 那一行，寫在〈host-check.py（3fc41c0）的已知限制與重現方式〉的〈指令的執行與 exit code：沒有接住的例外〉。</li>
<li><strong>結果檔的「目錄不存在時以 0700 建立」</strong>：0700 只套在 <code>os.makedirs</code> 這次新建的最後一層目錄，中間各層用預設的權限，寫在〈結果檔的讀寫與跟前一次的差異〉。</li>
<li><strong>只讀保證的「檔案大小與時間用 Python 的 os.lstat 量」</strong>：會累積的資料用 <code>os.lstat</code>；<code>check_logs</code> 確認檔案型的 log 時用 <code>os.stat</code>（會跟隨符號連結），檔案系統的剩餘空間用 <code>os.statvfs</code>，寫在〈定期讀的 log〉與〈檔案系統的剩餘空間〉。</li>
</ul>
<p><code>WHERE_VALUES</code> 到 <code>PHASES</code> 是清單欄位允許的值，下面的 <code>validate</code> 拿它們比對每個欄位，<code>PHASES</code> 同時是命令列 <code>--phase</code> 的選項。這幾個常數只是允許值的清單：加一個值之後，<code>validate</code> 就放行它，而用到那個值的分支沒有跟著改時，新的值在執行時出錯或被當成別的值處理。每個常數要一起改的位置：</p>
<ul>
<li><strong><code>WHERE_VALUES</code></strong>：<code>_manual_text</code> 用 <code>{&quot;console&quot;: &quot;主控台&quot;, &quot;external&quot;: &quot;主機外&quot;}</code> 查表，新的值在印 <code>MANUAL</code> 時丟出 <code>KeyError</code>（實跑：<code>_manual_text(&quot;bastion&quot;, …)</code> 丟出 <code>KeyError: 'bastion'</code>）；<code>validate</code> 與 <code>Checker</code> 都只分 <code>where == &quot;host&quot;</code> 與其餘，其餘的值一律走主機外的路徑（讀碼）。</li>
<li><strong><code>MEASURE_VALUES</code></strong>：<code>Checker.measure</code> 只分 <code>lines</code> 與 <code>value</code>，其餘一律當成時間解析；新的量法沒有在 <code>measure</code> 加分支時，被當成 <code>age_days</code> 算出天數（實跑：<code>measure(&quot;bytes&quot;, &quot;x=2026-10-26T00:00:00Z&quot;)</code> 回傳了一個天數）。</li>
<li><strong><code>PHASES</code></strong>：<code>Checker.run</code> 把 <code>maintain</code> 以外的值都交給 <code>check_phase</code>、讀 <code>self.m[phase]</code>，而 <code>validate</code> 只為 <code>for phase in (&quot;suspend&quot;, &quot;resume&quot;)</code> 這兩個建段落；新的階段要同時改這個迴圈，否則執行時丟出 <code>KeyError</code>（實跑：<code>Checker({}).run(&quot;drain&quot;)</code> 丟出 <code>KeyError: 'drain'</code>）。</li>
<li><strong><code>PROTOCOL_VALUES</code></strong>（讀碼）：<code>LISTEN_COMMAND</code> 只帶 <code>-t -u</code>，<code>listening_sockets</code> 只收這兩種協定，<code>audit_listening</code> 的提示用協定的第一個字母組 <code>ss -lnp{p[0]}</code>；新的協定要同時改這三處，以及 <code>ss</code> 規則允許的選項。</li>
<li><strong><code>EXPOSURE_VALUES</code>、<code>BILLED_VALUES</code></strong>（讀碼）：只在 <code>validate</code> 比對、在理由裡原樣印出，加值不必改其他位置。<code>COMMAND_TIMEOUT_S</code> 是 <code>Runner</code> 等一條指令結束的上限，超過時那一項記成 exit 124 並印成 <code>WARN</code>：一條卡住的指令（等網路、等鎖）只讓那一項失敗，其餘檢查照常跑完。<code>DEFAULT_STATE_FILE</code> 與 <code>STATE_FORMAT</code> 是結果檔的預設路徑與格式版本。<code>VAR_RE</code> 比對清單字串裡的 <code>{{名稱}}</code>，名稱只收 ASCII 的英文字母、數字與底線，第一個字元不能是數字；Python 的識別字另外收非 ASCII 的字母，<code>{{主機}}</code> 這種名稱在這裡對不上，原樣留在字串裡。</li>
</ul>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">1</span><span class="cl"><span class="k">class</span> <span class="nc">ManifestError</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">    <span class="s2">&#34;&#34;&#34;清單格式錯；errors 是逐條的原因。&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl">
</span></span><span class="line"><span class="ln">4</span><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">errors</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]):</span>
</span></span><span class="line"><span class="ln">5</span><span class="cl">        <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">errors</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">6</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">errors</span> <span class="o">=</span> <span class="n">errors</span></span></span></code></pre></div><p><code>ManifestError</code> 帶著逐條的錯誤原因。驗證清單的程式把整份清單的錯誤都收齊之後才丟一次這個例外，命令列再逐行印出 <code>manifest error:</code>，所以改清單的人一次看到全部的錯誤，不必改一條、跑一次。用 Ansible 部署時，<code>template</code> 任務的 <code>validate</code> 在受管主機上載入這支腳本、呼叫 <code>load_manifest</code> 驗證剛產生的清單，清單格式錯時這個例外讓那個任務失敗，格式錯的清單不會寫到正式的位置，做法在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈部署清單：建目錄、copy 與 template、validate 改用檢查腳本的載入函式、&ndash;check 的預覽〉。</p>
<h2 id="選項的切分optionspec-與-getopt_long-語法">選項的切分：OptionSpec 與 getopt_long 語法</h2>
<p>白名單要比對一條指令的選項，得先分出一串參數裡哪些是選項、哪些是選項的值、哪些是位置參數。這一段照 <code>getopt_long</code>（GNU C 函式庫解析命令列選項的函式，多數 Linux 工具照它的規則解析參數）的語法切分，<code>systemctl</code>、<code>journalctl</code>、<code>ss</code>、<code>needs-restarting</code>、<code>dnf</code>、<code>rpm</code> 的規則共用它。這六支裡，<code>systemctl</code>、<code>journalctl</code>、<code>ss</code> 用 <code>getopt_long</code> 解析參數，<code>rpm</code> 用 popt，<code>dnf</code> 與 <code>needs-restarting</code> 用 Python 的 argparse；popt 與 argparse 在短選項合併、值黏在選項後面、<code>--名稱=值</code> 與 <code>--</code> 這幾點，切法跟 <code>getopt_long</code> 相同，所以共用同一套切分；為什麼要照工具自己的語法切、沒照語法切時放行了哪些寫法，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># ── 只讀白名單 ──────────────────────────────────────────────</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"># 每條規則收 argv（argv[0] 已是 basename），回傳 None 代表放行，字串代表拒絕的理由。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="c1"># 一支工具允許的選項（列允許，表外一律拒絕）；選項名連同開頭的 - 一起寫。</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="nd">@dataclass</span><span class="p">(</span><span class="n">frozen</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="k">class</span> <span class="nc">OptionSpec</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="n">flags</span><span class="p">:</span> <span class="nb">frozenset</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">()</span>  <span class="c1"># 不吃值的旗標</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="n">values</span><span class="p">:</span> <span class="nb">frozenset</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">()</span>  <span class="c1"># 一定吃值的選項</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="n">optional</span><span class="p">:</span> <span class="nb">frozenset</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">()</span>  <span class="c1"># 選填值：只吃黏在一起的值，或下一個字是整數時</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">
</span></span><span class="line"><span class="ln">12</span><span class="cl">
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="k">def</span> <span class="nf">_spec</span><span class="p">(</span><span class="n">flags</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="n">values</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="n">optional</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&#34;&#34;</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">OptionSpec</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">return</span> <span class="n">OptionSpec</span><span class="p">(</span><span class="nb">frozenset</span><span class="p">(</span><span class="n">flags</span><span class="o">.</span><span class="n">split</span><span class="p">()),</span> <span class="nb">frozenset</span><span class="p">(</span><span class="n">values</span><span class="o">.</span><span class="n">split</span><span class="p">()),</span> <span class="nb">frozenset</span><span class="p">(</span><span class="n">optional</span><span class="o">.</span><span class="n">split</span><span class="p">()))</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">
</span></span><span class="line"><span class="ln">16</span><span class="cl">
</span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="k">def</span> <span class="nf">_parse_options</span><span class="p">(</span><span class="n">args</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">spec</span><span class="p">:</span> <span class="n">OptionSpec</span><span class="p">,</span> <span class="n">stop_at_first</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="nb">set</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="c1"># 依 getopt_long 的語法把 args 切成（位置參數, 出現過的選項, 拒絕理由）：</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">    <span class="c1"># 短選項可合併（-qa），吃值的短選項值黏在後面或在下一個字；長選項 --name[=value]；-- 之後全是位置參數。</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">    <span class="c1"># stop_at_first 時，第一個位置參數起的全部字都放進位置參數（不再解析選項）。</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="n">positionals</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">    <span class="n">seen</span><span class="p">:</span> <span class="nb">set</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="n">tok</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">        <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="k">if</span> <span class="n">tok</span> <span class="o">==</span> <span class="s2">&#34;--&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">            <span class="n">positionals</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">:])</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">            <span class="k">break</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">tok</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;-&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">tok</span> <span class="o">==</span> <span class="s2">&#34;-&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">            <span class="n">positionals</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">tok</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">            <span class="k">if</span> <span class="n">stop_at_first</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">                <span class="n">positionals</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">:])</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">                <span class="k">break</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">        <span class="k">if</span> <span class="n">tok</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;--&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">            <span class="n">name</span><span class="p">,</span> <span class="n">eq</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">tok</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;=&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">            <span class="n">consumed</span> <span class="o">=</span> <span class="n">_long_option</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="nb">bool</span><span class="p">(</span><span class="n">eq</span><span class="p">),</span> <span class="n">spec</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">            <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">consumed</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">                <span class="k">return</span> <span class="n">positionals</span><span class="p">,</span> <span class="n">seen</span><span class="p">,</span> <span class="n">consumed</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">            <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">            <span class="n">i</span> <span class="o">+=</span> <span class="n">consumed</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">        <span class="k">for</span> <span class="n">pos</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">tok</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">45</span><span class="cl">            <span class="n">name</span> <span class="o">=</span> <span class="s2">&#34;-&#34;</span> <span class="o">+</span> <span class="n">tok</span><span class="p">[</span><span class="n">pos</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">            <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">spec</span><span class="o">.</span><span class="n">flags</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">47</span><span class="cl">                <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">48</span><span class="cl">                <span class="k">continue</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">            <span class="n">consumed</span> <span class="o">=</span> <span class="n">_value_option</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">pos</span> <span class="o">+</span> <span class="mi">1</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">tok</span><span class="p">),</span> <span class="n">spec</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">            <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">consumed</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">                <span class="k">return</span> <span class="n">positionals</span><span class="p">,</span> <span class="n">seen</span><span class="p">,</span> <span class="n">consumed</span>
</span></span><span class="line"><span class="ln">52</span><span class="cl">            <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">53</span><span class="cl">            <span class="n">i</span> <span class="o">+=</span> <span class="n">consumed</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">            <span class="k">break</span>
</span></span><span class="line"><span class="ln">55</span><span class="cl">    <span class="k">return</span> <span class="n">positionals</span><span class="p">,</span> <span class="n">seen</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">56</span><span class="cl">
</span></span><span class="line"><span class="ln">57</span><span class="cl">
</span></span><span class="line"><span class="ln">58</span><span class="cl"><span class="k">def</span> <span class="nf">_long_option</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">has_value</span><span class="p">:</span> <span class="nb">bool</span><span class="p">,</span> <span class="n">spec</span><span class="p">:</span> <span class="n">OptionSpec</span><span class="p">,</span> <span class="n">args</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">i</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span> <span class="o">|</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">59</span><span class="cl">    <span class="c1"># 回傳這個長選項多吃掉幾個後續字；拒絕時回傳理由。</span>
</span></span><span class="line"><span class="ln">60</span><span class="cl">    <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">spec</span><span class="o">.</span><span class="n">flags</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">61</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;選項 </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2"> 不吃值&#34;</span> <span class="k">if</span> <span class="n">has_value</span> <span class="k">else</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">62</span><span class="cl">    <span class="k">return</span> <span class="n">_value_option</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">has_value</span><span class="p">,</span> <span class="n">spec</span><span class="p">,</span> <span class="n">args</span><span class="p">,</span> <span class="n">i</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">63</span><span class="cl">
</span></span><span class="line"><span class="ln">64</span><span class="cl">
</span></span><span class="line"><span class="ln">65</span><span class="cl"><span class="k">def</span> <span class="nf">_value_option</span><span class="p">(</span><span class="n">name</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">has_glued_value</span><span class="p">:</span> <span class="nb">bool</span><span class="p">,</span> <span class="n">spec</span><span class="p">:</span> <span class="n">OptionSpec</span><span class="p">,</span> <span class="n">args</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="n">i</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span> <span class="o">|</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">66</span><span class="cl">    <span class="k">if</span> <span class="n">name</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">spec</span><span class="o">.</span><span class="n">values</span> <span class="ow">and</span> <span class="n">name</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">spec</span><span class="o">.</span><span class="n">optional</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">67</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;選項 </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2"> 不在允許清單&#34;</span>
</span></span><span class="line"><span class="ln">68</span><span class="cl">    <span class="k">if</span> <span class="n">has_glued_value</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">69</span><span class="cl">        <span class="k">return</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">70</span><span class="cl">    <span class="n">nxt</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="k">if</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="k">else</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">71</span><span class="cl">    <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">spec</span><span class="o">.</span><span class="n">optional</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">72</span><span class="cl">        <span class="k">return</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">nxt</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">nxt</span><span class="o">.</span><span class="n">isdigit</span><span class="p">()</span> <span class="k">else</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">73</span><span class="cl">    <span class="k">return</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">nxt</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="sa">f</span><span class="s2">&#34;選項 </span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2"> 缺少值&#34;</span></span></span></code></pre></div><p><code>OptionSpec</code> 把一支工具允許的選項分成三組，分組決定切分時這個選項是否吃掉下一個參數：<code>flags</code> 不吃值；<code>values</code> 一定吃值，值黏在選項後面（<code>-pX</code>）時吃黏著的那一段，否則吃下一個參數；<code>optional</code> 是值可帶可不帶的選項，值黏在選項後面時吃黏著的那一段，沒有黏著的值時，下一個參數全是數字才吃它，否則不吃。<code>journalctl</code> 的 <code>-n</code> 放在 <code>optional</code>，所以 <code>journalctl -n 20 -u sshd</code> 的 <code>20</code> 被當成 <code>-n</code> 的值，<code>journalctl -n -u sshd</code> 的 <code>-u</code> 照樣被當成選項比對。「全是數字」用 <code>str.isdigit()</code> 判斷：<code>-5</code>、<code>+5</code> 帶著正負號，不算數字，不會被當成值；<code>²</code>、<code>٣</code> 這類 Unicode 的數字字元，<code>isdigit()</code> 回 <code>True</code>，會被當成值（這幾個字串都在 Python 裡實跑過）。把一個吃值的選項錯放進 <code>flags</code>，它的值會被當成下一個位置參數，子命令的判定就跟工具實際執行的不同。</p>
<p>短選項的那個迴圈從第二個字元起逐字查：查到旗標就看下一個字元（<code>-qa</code> 是 <code>-q</code> 與 <code>-a</code>）；查到吃值的選項，這個字元後面還有字時（<code>-ppid</code>）把剩下的當成值，沒有時吃下一個參數，然後離開這個參數。單獨一個 <code>-</code> 算位置參數：照 <code>getopt</code> 的慣例，它是一個運算元（多半代表標準輸入），不是選項。長選項用 <code>partition(&quot;=&quot;)</code> 分出名稱與值，旗標帶了 <code>=值</code> 時拒絕（<code>--no-pager=1</code>）。名稱一律整字比對，<code>getopt_long</code> 接受的前綴縮寫（<code>--cursor-fi</code> 當成 <code>--cursor-file</code>）不在表上，所以被拒絕。表外的選項回傳拒絕理由、不往下切，三組都不在表上的選項就是被拒絕的那一類，這是列允許的寫法在程式裡的樣子。</p>
<p><code>stop_at_first</code> 給 <code>dnf</code> 用：切到第一個位置參數（子命令）就停，子命令之後的參數交給那個子命令自己的規則。單元測試 <code>test_allowed</code>（本篇沒有收錄）放行 <code>systemctl --no-pager -ppid status nginx</code>；<code>test_rejected</code> 拒絕 <code>systemctl -p show stop nginx</code>（<code>-p</code> 吃掉 <code>show</code>，子命令是 <code>stop</code>）、<code>systemctl -pshow stop nginx</code> 與 <code>systemctl --no-pager=1 status nginx</code>。</p>
<h2 id="各工具的選項規則systemctljournalctlfirewall-cmdssfindsqlite3opensslsshdneeds-restartingdnfrpmausearch">各工具的選項規則：systemctl、journalctl、firewall-cmd、ss、find、sqlite3、openssl、sshd、needs-restarting、dnf、rpm、ausearch</h2>
<p>每支有選項規則的工具一個函式，收切好的參數串列（第一個元素已經換成檔名），放行時回傳 <code>None</code>，拒絕時回傳理由的字串。每支工具允許與排除了哪些選項、排除的理由，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉的各工具清單；這一節寫程式怎麼實現那些規則，以及每一支的語法跟〈選項的切分：OptionSpec 與 getopt_long 語法〉的切分有哪裡不同。</p>
<h3 id="systemctl-的子命令與選項">systemctl 的子命令與選項</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="n">SYSTEMCTL_READ</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;is-active&#34;</span><span class="p">,</span> <span class="s2">&#34;is-enabled&#34;</span><span class="p">,</span> <span class="s2">&#34;is-failed&#34;</span><span class="p">,</span> <span class="s2">&#34;show&#34;</span><span class="p">,</span> <span class="s2">&#34;status&#34;</span><span class="p">,</span> <span class="s2">&#34;cat&#34;</span><span class="p">,</span> <span class="s2">&#34;list-timers&#34;</span><span class="p">,</span> <span class="s2">&#34;list-units&#34;</span><span class="p">,</span> <span class="s2">&#34;list-unit-files&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">SYSTEMCTL_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="n">flags</span><span class="o">=</span><span class="s2">&#34;--no-pager --no-legend --plain --value -a --all -l --full -q --quiet --failed&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="n">values</span><span class="o">=</span><span class="s2">&#34;-p --property -P -t --type --state -n --lines -o --output --timestamp&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_systemctl</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="c1"># -p、-t 這類選項會吃掉下一個字，所以子命令要先依選項語法切出來，不能取第一個不以 - 開頭的字。</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="c1"># -H/--host 與 -M/--machine 會把操作送到別台主機或容器（-H 以 root 的 SSH 身分連出去），不在允許清單。</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="n">positionals</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">SYSTEMCTL_OPTIONS</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="k">if</span> <span class="n">err</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;systemctl </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">positionals</span> <span class="ow">or</span> <span class="n">positionals</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">SYSTEMCTL_READ</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;systemctl 只允許 </span><span class="si">{</span><span class="nb">sorted</span><span class="p">(</span><span class="n">SYSTEMCTL_READ</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p>子命令取切分之後的第一個位置參數，查 <code>SYSTEMCTL_READ</code>。只寫 <code>systemctl</code> 而沒有子命令時，<code>positionals</code> 是空的，同樣拒絕：systemctl 沒有子命令時執行的是 <code>list-units</code>，規則要求清單把子命令寫出來，判定才不依賴工具的預設值。<code>-H</code> 與 <code>-M</code> 不在 <code>SYSTEMCTL_OPTIONS</code> 裡，所以黏在一起的 <code>-Ha@b</code>、長選項的 <code>--host=a@b</code> 與分開寫的 <code>--host a@b</code> 都在切分時被拒絕，單元測試 <code>test_rejected</code>（本篇沒有收錄）逐一列了這幾種寫法。</p>
<h3 id="journalctl-的選項與-match-條件">journalctl 的選項與 match 條件</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># --cursor-file 與 --vacuum-*、--rotate、--flush、-f 都會寫檔或改變 journal，不在允許清單。</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">JOURNALCTL_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="n">flags</span><span class="o">=</span><span class="s2">&#34;-q --quiet --no-pager -r --reverse --utc --no-hostname -k --dmesg -a --all --disk-usage --list-boots&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="n">values</span><span class="o">=</span><span class="s2">&#34;-u --unit --since --until -o --output -g --grep -p --priority -t --identifier&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="n">optional</span><span class="o">=</span><span class="s2">&#34;-n --lines&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_journalctl</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">JOURNALCTL_OPTIONS</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;journalctl </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="n">err</span> <span class="k">else</span> <span class="kc">None</span></span></span></code></pre></div><p><code>journalctl</code> 的位置參數是篩選紀錄的 match 條件（<code>_SYSTEMD_UNIT=x.service</code>），只用來讀，所以規則只比對選項、不看位置參數。<code>-n</code> 放在 <code>optional</code>，<code>test_rejected</code> 的 <code>journalctl -n --vacuum-size=1G</code> 因此被拒絕：<code>-n</code> 不吃以 <code>-</code> 開頭的下一個參數，<code>--vacuum-size</code> 照樣被當成選項比對，而它不在表上。<code>-f</code>（持續讀新的紀錄、不會結束）也不在表上，它會讓指令一直跑到 <code>COMMAND_TIMEOUT_S</code> 的上限。</p>
<h3 id="firewall-cmd-的參數樣式">firewall-cmd 的參數樣式</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">1</span><span class="cl"><span class="n">FIREWALL_READ</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;^--(permanent|zone=\S+|state|query-\S+|list-\S+|get-\S+|info-\S+)$&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">
</span></span><span class="line"><span class="ln">3</span><span class="cl">
</span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_firewall_cmd</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">5</span><span class="cl">    <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
</span></span><span class="line"><span class="ln">6</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">FIREWALL_READ</span><span class="o">.</span><span class="k">match</span><span class="p">(</span><span class="n">a</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">7</span><span class="cl">            <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;firewall-cmd 只允許 --query-*、--list-*、--get-*、--info-*、--state、--permanent、--zone=，不允許 </span><span class="si">{</span><span class="n">a</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">8</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>firewall-cmd</code> 的查詢都寫成一個 <code>--名稱</code> 或 <code>--名稱=值</code> 的參數，所以規則逐個參數比對樣式，不切分。每個參數都要對上樣式，位置參數因此一個都不允許：分開寫的 <code>--zone public</code> 裡，<code>public</code> 對不上任何樣式而被拒絕，值要用 <code>=</code> 黏在選項上。改動設定的選項（<code>--add-port=</code>、<code>--remove-*</code>、<code>--reload</code>）對不上 <code>query-</code>、<code>list-</code>、<code>get-</code>、<code>info-</code> 這幾個前綴，<code>test_rejected</code> 列了 <code>firewall-cmd --add-port=8765/tcp</code> 與 <code>firewall-cmd --reload</code>。</p>
<h3 id="ss-的選項與過濾式">ss 的選項與過濾式</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># -K（kill）、-D（寫檔）、-N（換 netns）、-F（讀過濾檔）、-r（反查 DNS，對外連線）、-E 都不在允許清單。</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"># 位置參數是過濾式（state listening、sport = :443 這類），ss 只讀它。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="n">SS_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="n">flags</span><span class="o">=</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">        <span class="s2">&#34;-H --no-header -l --listening -n --numeric -t --tcp -u --udp -a --all -p --processes -4 --ipv4 -6 --ipv6 &#34;</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">        <span class="s2">&#34;-x --unix -w --raw -e --extended -i --info -m --memory -s --summary -o --options&#34;</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="p">),</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="n">values</span><span class="o">=</span><span class="s2">&#34;-f --family -A --query&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">
</span></span><span class="line"><span class="ln">11</span><span class="cl">
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_ss</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">SS_OPTIONS</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;ss </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="n">err</span> <span class="k">else</span> <span class="kc">None</span></span></span></code></pre></div><p><code>ss</code> 的位置參數是過濾式，跟 <code>journalctl</code> 一樣只比對選項。合併的短選項逐字查，所以 <code>ss -tlnK</code> 在查到 <code>K</code> 時被拒絕，<code>-K</code> 不必單獨出現；<code>-f</code> 一定吃值，單獨的 <code>ss -f</code> 以「缺少值」拒絕。<code>-E</code>（持續印出 socket 的事件、不會結束）不在表上，理由跟 <code>journalctl -f</code> 相同。</p>
<h3 id="find-的運算式">find 的運算式</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># find 不是 getopt 語法，所以自己走一遍：選項只認下面兩張表（-delete、-exec*、-ok*、-fprint*、-fls、</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"># -D、-files0-from 都不在裡面）；值只吃下一個字；不以 - 開頭的字是路徑或運算子的括號、驚嘆號。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="n">FIND_LEADING</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s2">&#34;-H&#34;</span><span class="p">,</span> <span class="s2">&#34;-L&#34;</span><span class="p">,</span> <span class="s2">&#34;-P&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="n">FIND_OPERATORS</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s2">&#34;(&#34;</span><span class="p">,</span> <span class="s2">&#34;)&#34;</span><span class="p">,</span> <span class="s2">&#34;!&#34;</span><span class="p">,</span> <span class="s2">&#34;,&#34;</span><span class="p">,</span> <span class="s2">&#34;-not&#34;</span><span class="p">,</span> <span class="s2">&#34;-and&#34;</span><span class="p">,</span> <span class="s2">&#34;-or&#34;</span><span class="p">,</span> <span class="s2">&#34;-a&#34;</span><span class="p">,</span> <span class="s2">&#34;-o&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="n">FIND_FLAGS</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    <span class="s2">&#34;-print -print0 -prune -empty -xdev -mount -true -false -readable -writable -executable&#34;</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="n">FIND_VALUES</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="p">(</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">        <span class="s2">&#34;-name -iname -path -ipath -regex -iregex -type -xtype -mtime -mmin -atime -amin -ctime -cmin &#34;</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">        <span class="s2">&#34;-newer -size -user -group -uid -gid -perm -links -inum -maxdepth -mindepth -printf&#34;</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">
</span></span><span class="line"><span class="ln">15</span><span class="cl">
</span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_find</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="n">args</span> <span class="o">=</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="ow">and</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="ow">in</span> <span class="n">FIND_LEADING</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">        <span class="n">tok</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">        <span class="k">if</span> <span class="n">tok</span> <span class="ow">in</span> <span class="n">FIND_FLAGS</span> <span class="ow">or</span> <span class="n">tok</span> <span class="ow">in</span> <span class="n">FIND_OPERATORS</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">        <span class="k">if</span> <span class="n">tok</span> <span class="ow">in</span> <span class="n">FIND_VALUES</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">            <span class="k">if</span> <span class="n">i</span> <span class="o">&gt;=</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">                <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;find 的選項 </span><span class="si">{</span><span class="n">tok</span><span class="si">}</span><span class="s2"> 缺少值&#34;</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">            <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">        <span class="k">if</span> <span class="n">tok</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;-&#34;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">tok</span> <span class="o">!=</span> <span class="s2">&#34;-&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">            <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;find 的選項 </span><span class="si">{</span><span class="n">tok</span><span class="si">}</span><span class="s2"> 不在允許清單&#34;</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>find</code> 的參數是起點路徑加一串運算式，運算式的元素是單一連字號的整字（<code>-name</code>、<code>-mtime</code>），不能照 <code>getopt_long</code> 合併或拆字，所以規則自己逐個參數比對。<code>-H</code>、<code>-L</code>、<code>-P</code> 只在最前面接受，跟 <code>find</code> 自己的語法一致；放在中間的 <code>-L</code> 對不上任何一張表而被拒絕。以 <code>-</code> 開頭而不在表上的參數一律拒絕，<code>-delete</code>、<code>-exec</code>、<code>-fprint</code> 這類動作因此不必逐一列出來擋；<code>-newerXY</code> 這種帶後綴的寫法也不在表上，<code>test_rejected</code> 列了它。<code>-printf</code> 在允許的表上，因為它把格式化的結果印到標準輸出；同類的 <code>-fprintf</code> 寫進檔案，不在表上。</p>
<h3 id="sqlite3-的必帶選項與-dot-command">sqlite3 的必帶選項與 dot-command</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># sqlite3 的長選項是單破折號整字（--x 等同 -x），不是 getopt 語法，所以自己走一遍：先把開頭的 -- 正規化成 -，再整字比對。</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">SQLITE_FLAGS</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="s2">&#34;-readonly -safe -bail -batch -header -noheader -csv -json -line -list -box -table -markdown -column -quote -tabs -nofollow&#34;</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="n">SQLITE_VALUES</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span><span class="s2">&#34;-separator -newline -nullvalue&#34;</span><span class="o">.</span><span class="n">split</span><span class="p">())</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="n">SQLITE_REQUIRED</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s2">&#34;-readonly&#34;</span><span class="p">,</span> <span class="s2">&#34;-safe&#34;</span><span class="p">,</span> <span class="s2">&#34;-init&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_sqlite3</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="c1"># -readonly 只讓資料庫檔唯讀；SQL 函式 writefile()、edit() 與 ATTACH 仍能寫檔案系統。</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="c1"># -safe 擋下這些函式與會寫檔的 dot-command（OL10 的 3.46.1 實測支援）。</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="c1"># -init 檔與 ~/.sqliterc 在 safe mode 生效前就執行（.shell 照跑，實測會建檔），HOME 擋不住</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="c1"># （sqlite3 用 getpwuid 找家目錄）；唯一的擋法是 -init /dev/null 取代 ~/.sqliterc。三個都要帶。</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="n">args</span> <span class="o">=</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">    <span class="n">seen</span><span class="p">:</span> <span class="nb">set</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="n">positionals</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="n">tok</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">tok</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;-&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">            <span class="n">positionals</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">tok</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">        <span class="n">name</span> <span class="o">=</span> <span class="n">tok</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="k">if</span> <span class="n">tok</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;--&#34;</span><span class="p">)</span> <span class="k">else</span> <span class="n">tok</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">SQLITE_FLAGS</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">            <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="k">elif</span> <span class="n">name</span> <span class="o">==</span> <span class="s2">&#34;-init&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">            <span class="k">if</span> <span class="n">i</span> <span class="o">&gt;=</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="ow">or</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">!=</span> <span class="s2">&#34;/dev/null&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">                <span class="k">return</span> <span class="s2">&#34;sqlite3 的 -init 的值必須是 /dev/null&#34;</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">            <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">            <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">        <span class="k">elif</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">SQLITE_VALUES</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">            <span class="k">if</span> <span class="n">i</span> <span class="o">&gt;=</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">                <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;sqlite3 的選項 </span><span class="si">{</span><span class="n">tok</span><span class="si">}</span><span class="s2"> 缺少值&#34;</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">            <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">            <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;sqlite3 的選項 </span><span class="si">{</span><span class="n">tok</span><span class="si">}</span><span class="s2"> 不在允許清單&#34;</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">SQLITE_REQUIRED</span> <span class="o">&lt;=</span> <span class="n">seen</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34;sqlite3 要同時帶 -init /dev/null、-readonly 與 -safe（~/.sqliterc 與 writefile() 都會寫檔）&#34;</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">    <span class="k">for</span> <span class="n">sql</span> <span class="ow">in</span> <span class="n">positionals</span><span class="p">[</span><span class="mi">1</span><span class="p">:]:</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">        <span class="k">if</span> <span class="nb">any</span><span class="p">(</span><span class="n">line</span><span class="o">.</span><span class="n">lstrip</span><span class="p">()</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;.&#34;</span><span class="p">)</span> <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">sql</span><span class="o">.</span><span class="n">splitlines</span><span class="p">()):</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">            <span class="k">return</span> <span class="s2">&#34;sqlite3 的 dot-command（.backup、.output、.shell 這類）不允許&#34;</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>sqlite3</code> 的選項是單一連字號的整字，雙連字號的寫法（<code>--readonly</code>）跟單一連字號的意思相同，所以規則先把開頭的 <code>--</code> 換成 <code>-</code> 再比對，<code>test_sqlite_requires_init_devnull</code> 放行 <code>sqlite3 --init /dev/null --readonly --safe -json …</code>。三個必帶的選項用集合的子集比對（<code>SQLITE_REQUIRED &lt;= seen</code>），少任何一個都拒絕；<code>-init</code> 的值只接受 <code>/dev/null</code>，<code>-init -readonly -safe</code> 這種把下一個選項當成值的寫法也被拒絕，因為 <code>-readonly</code> 不等於 <code>/dev/null</code>。為什麼三個都要帶，在清單寫成 TOML 的那篇概念文章的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉講 <code>sqlite3</code> 執行 shell 與寫檔途徑的那一段。</p>
<p>位置參數裡，第一個是資料庫檔，之後的每一個是要執行的 SQL。規則把每段 SQL 逐行看，去掉行首空白之後以 <code>.</code> 開頭的行就是 dot-command，一律拒絕，不只靠 <code>-safe</code> 擋。在 SQL 參數之外塞 dot-command 的 <code>-cmd</code> 選項不在表上；<code>-A</code>（操作 SQLite 封存檔）、<code>-interactive</code>、<code>-echo</code>、<code>-vfs</code> 也不在表上，<code>test_sqlite_requires_init_devnull</code> 逐一列了它們。程式碼註解裡的「OL10 的 3.46.1」指 Oracle Linux 10 上的 sqlite 3.46.1。</p>
<h3 id="openssl-與-sshd-的固定寫法">openssl 與 sshd 的固定寫法</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="n">OPENSSL_X509_OPTS</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;-noout&#34;</span><span class="p">,</span> <span class="s2">&#34;-in&#34;</span><span class="p">,</span> <span class="s2">&#34;-checkend&#34;</span><span class="p">,</span> <span class="s2">&#34;-enddate&#34;</span><span class="p">,</span> <span class="s2">&#34;-startdate&#34;</span><span class="p">,</span> <span class="s2">&#34;-dates&#34;</span><span class="p">,</span> <span class="s2">&#34;-subject&#34;</span><span class="p">,</span> <span class="s2">&#34;-issuer&#34;</span><span class="p">,</span> <span class="s2">&#34;-serial&#34;</span><span class="p">,</span> <span class="s2">&#34;-fingerprint&#34;</span><span class="p">,</span> <span class="s2">&#34;-sha256&#34;</span><span class="p">,</span> <span class="s2">&#34;-ext&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_openssl</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">2</span> <span class="ow">or</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">!=</span> <span class="s2">&#34;x509&#34;</span> <span class="ow">or</span> <span class="s2">&#34;-noout&#34;</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">argv</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34;openssl 只允許 x509 -noout&#34;</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">argv</span><span class="p">[</span><span class="mi">2</span><span class="p">:]:</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">        <span class="k">if</span> <span class="n">a</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;-&#34;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">a</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">OPENSSL_X509_OPTS</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">            <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;openssl x509 不允許 </span><span class="si">{</span><span class="n">a</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>openssl</code> 只允許 <code>x509</code> 子命令，而且一定要帶 <code>-noout</code>（不印出憑證本身的編碼內容）；其餘以 <code>-</code> 開頭的參數要在 <code>OPENSSL_X509_OPTS</code> 裡。不以 <code>-</code> 開頭的參數是選項的值（<code>-in</code> 的憑證路徑、<code>-checkend</code> 的秒數），放行。寫出檔案的 <code>-out</code> 不在表上，<code>test_rejected</code> 列了 <code>openssl x509 -noout -in a -out b</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">1</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_sshd</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">    <span class="c1"># -E 與 -f 會寫檔或換設定，所以只接受單獨的 -T。</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span> <span class="k">if</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span> <span class="o">==</span> <span class="p">[</span><span class="s2">&#34;-T&#34;</span><span class="p">]</span> <span class="k">else</span> <span class="s2">&#34;sshd 只允許單獨的 -T（印出生效中的設定）&#34;</span></span></span></code></pre></div><p><code>sshd</code> 的規則把參數串列跟 <code>[&quot;-T&quot;]</code> 整串比對：檢查要的只有「印出生效中的設定」這一種用法，整串比對比列出允許的選項更窄，<code>-T -E &lt;檔案&gt;</code> 與 <code>-E &lt;檔案&gt; -T</code> 兩種順序都被拒絕。</p>
<h3 id="needs-restarting-與-dnf-的子命令">needs-restarting 與 dnf 的子命令</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="n">NEEDS_RESTARTING_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">(</span><span class="n">flags</span><span class="o">=</span><span class="s2">&#34;-r -s --reboothint --services&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_needs_restarting</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="n">positionals</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">NEEDS_RESTARTING_OPTIONS</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    <span class="k">if</span> <span class="n">err</span> <span class="ow">or</span> <span class="n">positionals</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34;needs-restarting 只允許 -r、-s、--reboothint、--services&#34;</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">
</span></span><span class="line"><span class="ln">10</span><span class="cl">
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="n">DNF_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">(</span><span class="n">flags</span><span class="o">=</span><span class="s2">&#34;-q --quiet -C --cacheonly&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="n">NO_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">
</span></span><span class="line"><span class="ln">14</span><span class="cl">
</span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_dnf</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="c1"># -x、--setopt、-c、--installroot 這類吃值的選項不在允許清單；只允許子命令前的 -q、-C。</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="n">rest</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">DNF_OPTIONS</span><span class="p">,</span> <span class="n">stop_at_first</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="k">if</span> <span class="n">err</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;dnf </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">    <span class="k">if</span> <span class="n">rest</span><span class="p">[:</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="p">[</span><span class="s2">&#34;needs-restarting&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">        <span class="k">return</span> <span class="n">_rule_needs_restarting</span><span class="p">(</span><span class="n">rest</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">    <span class="k">if</span> <span class="n">rest</span><span class="p">[:</span><span class="mi">2</span><span class="p">]</span> <span class="ow">in</span> <span class="p">([</span><span class="s2">&#34;history&#34;</span><span class="p">,</span> <span class="s2">&#34;list&#34;</span><span class="p">],</span> <span class="p">[</span><span class="s2">&#34;history&#34;</span><span class="p">,</span> <span class="s2">&#34;info&#34;</span><span class="p">]):</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">rest</span><span class="p">[</span><span class="mi">2</span><span class="p">:],</span> <span class="n">NO_OPTIONS</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;dnf </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="n">err</span> <span class="k">else</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">    <span class="k">return</span> <span class="s2">&#34;dnf 只允許 needs-restarting 與 history list/info&#34;</span></span></span></code></pre></div><p><code>dnf</code> 的規則分兩段切：子命令之前用 <code>DNF_OPTIONS</code> 切，切到第一個位置參數就停（<code>stop_at_first</code>）；剩下的參數以子命令開頭，交給子命令的規則。<code>needs-restarting</code> 這個子命令跟單獨呼叫的 <code>needs-restarting</code> 共用同一個規則函式，<code>rest</code> 的第一個元素正好是 <code>needs-restarting</code>，跟單獨呼叫時的參數串列形狀相同。<code>history list</code> 與 <code>history info</code> 之後用沒有任何允許選項的 <code>NO_OPTIONS</code> 切：位置參數（<code>dnf history info 12</code> 的交易編號）放行，任何選項都拒絕，<code>test_rejected</code> 的 <code>dnf history list -y</code> 就是這一種。<code>dnf -x history -x list install foo</code> 在子命令之前那一段就被拒絕，<code>-x</code> 不在 <code>DNF_OPTIONS</code> 裡。</p>
<h3 id="rpm-的查詢模式">rpm 的查詢模式</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># --eval、-E、-D/--define 展開巨集，%(...) 會經 shell 執行；--pipe 直接執行指令；</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"># --macros、--rcfile、--dbpath、--root、-p 換設定或資料來源。都不在允許清單。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="n">RPM_OPTIONS</span> <span class="o">=</span> <span class="n">_spec</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="n">flags</span><span class="o">=</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">        <span class="s2">&#34;-q -a -f -i -l -c -d -s -R --query --all --file --info --list --configfiles --docfiles &#34;</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">        <span class="s2">&#34;--state --requires --provides --changelog --last --whatprovides --whatrequires&#34;</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="p">),</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="n">values</span><span class="o">=</span><span class="s2">&#34;--qf --queryformat&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">
</span></span><span class="line"><span class="ln">11</span><span class="cl">
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_rpm</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="c1"># 第一個選項必須是查詢模式（-q、-qa、-qi 這類或 --query），不讓 -i、-U 這類模式排在前面。</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">2</span> <span class="ow">or</span> <span class="ow">not</span> <span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;-q&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;--query&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34;rpm 只允許查詢（第一個選項要是 -q 開頭或 --query）&#34;</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="n">_</span><span class="p">,</span> <span class="n">seen</span><span class="p">,</span> <span class="n">err</span> <span class="o">=</span> <span class="n">_parse_options</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:],</span> <span class="n">RPM_OPTIONS</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="k">if</span> <span class="n">err</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;rpm </span><span class="si">{</span><span class="n">err</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="p">({</span><span class="s2">&#34;-q&#34;</span><span class="p">,</span> <span class="s2">&#34;--query&#34;</span><span class="p">}</span> <span class="o">&amp;</span> <span class="n">seen</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34;rpm 只允許查詢（要帶 -q）&#34;</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>rpm</code> 的 <code>-i</code> 在查詢模式裡是「印出套件資訊」，排在最前面時是「安裝套件」，同一個字母的意思由模式決定。規則因此先要求第一個參數是查詢模式（<code>-q</code> 開頭或 <code>--query</code>），再照選項表切分：<code>RPM_OPTIONS</code> 裡的 <code>-i</code>、<code>-f</code> 只在查詢模式之後才出現得了，<code>test_rpm_first_option_is_query</code> 拒絕的 <code>rpm -i tree.rpm -q</code>、<code>rpm -iq</code>、<code>rpm -U x.rpm -q</code> 都在查第一個參數時被拒絕。切分之後再確認出現過的選項裡有 <code>-q</code> 或 <code>--query</code>：以現在的寫法，通過第一個參數檢查的輸入，切分之後一定帶著 <code>-q</code> 或 <code>--query</code>，這一道擋不下任何多出來的輸入（讀碼）；它是冗餘的防護，日後有人放寬第一個參數的比對時，查詢模式的要求不會跟著消失。<code>-qp</code>（查詢一個 <code>.rpm</code> 檔而不是已安裝的套件）的 <code>-p</code> 不在表上，<code>test_rejected</code> 列了 <code>rpm -qp /tmp/a.rpm</code>。</p>
<h3 id="ausearch-的篩選選項與時間">ausearch 的篩選選項與時間</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># ausearch 是單破折號多字母選項，整字比對。--checkpoint、-if/--input 這類寫檔或換輸入來源的選項不在允許清單。</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">AUSEARCH_FLAGS</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span><span class="s2">&#34;-i --interpret -r --raw --just-one&#34;</span><span class="o">.</span><span class="n">split</span><span class="p">())</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="n">AUSEARCH_VALUES</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="s2">&#34;-m --message -k --key -c --comm -p --pid -ui --uid -x --executable -sc --syscall -sv --success --format&#34;</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="n">AUSEARCH_TIME</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">(</span><span class="s2">&#34;-ts --start -te --end&#34;</span><span class="o">.</span><span class="n">split</span><span class="p">())</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="n">AUSEARCH_CLOCK</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;^\d{1,2}:\d</span><span class="si">{2}</span><span class="s2">(:\d</span><span class="si">{2}</span><span class="s2">)?$&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="k">def</span> <span class="nf">_rule_ausearch</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="n">args</span> <span class="o">=</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:]</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="k">while</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">        <span class="n">tok</span> <span class="o">=</span> <span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">        <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">        <span class="n">name</span><span class="p">,</span> <span class="n">eq</span><span class="p">,</span> <span class="n">_</span> <span class="o">=</span> <span class="n">tok</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;=&#34;</span><span class="p">)</span> <span class="k">if</span> <span class="n">tok</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;--&#34;</span><span class="p">)</span> <span class="k">else</span> <span class="p">(</span><span class="n">tok</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">        <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">AUSEARCH_FLAGS</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">eq</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">AUSEARCH_VALUES</span> <span class="ow">or</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">AUSEARCH_TIME</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">            <span class="k">if</span> <span class="ow">not</span> <span class="n">eq</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">                <span class="k">if</span> <span class="n">i</span> <span class="o">&gt;=</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">                    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;ausearch 的選項 </span><span class="si">{</span><span class="n">tok</span><span class="si">}</span><span class="s2"> 缺少值&#34;</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">                <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">                <span class="c1"># -ts 後面是日期時，再下一個字可能是時間（-ts 10/26/2026 10:30）</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">                <span class="k">if</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">AUSEARCH_TIME</span> <span class="ow">and</span> <span class="n">i</span> <span class="o">&lt;</span> <span class="nb">len</span><span class="p">(</span><span class="n">args</span><span class="p">)</span> <span class="ow">and</span> <span class="n">AUSEARCH_CLOCK</span><span class="o">.</span><span class="k">match</span><span class="p">(</span><span class="n">args</span><span class="p">[</span><span class="n">i</span><span class="p">]):</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">                    <span class="n">i</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;ausearch 的選項 </span><span class="si">{</span><span class="n">tok</span><span class="si">}</span><span class="s2"> 不在允許清單&#34;</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>ausearch</code> 的短選項有多個字母（<code>-ui</code>、<code>-sc</code>），不能照 <code>getopt_long</code> 拆字，所以整字比對；只有雙連字號的長選項接受 <code>=值</code>，單一連字號的 <code>-i=1</code> 整串對不上表而被拒絕。<code>-ts</code>、<code>-te</code> 的值可以是一個詞（<code>today</code>、<code>recent</code>），也可以是日期加時間兩個參數，所以吃掉值之後，下一個參數長得像時刻（<code>10:30</code> 或 <code>10:30:00</code>）時再吃一個。<code>test_ausearch_allow_list</code> 放行 <code>ausearch -ts 10/26/2026 10:30:00 -te 11:00</code>，拒絕 <code>-m</code> 沒有值、<code>--checkpoint</code>、<code>--input</code> 與 <code>-if</code>。</p>
<h3 id="不另外比對選項的工具與規則表">不另外比對選項的工具與規則表</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">def</span> <span class="nf">_allow</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="n">READONLY_RULES</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Callable</span><span class="p">[[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]],</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]]</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    <span class="s2">&#34;ss&#34;</span><span class="p">:</span> <span class="n">_rule_ss</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="s2">&#34;systemctl&#34;</span><span class="p">:</span> <span class="n">_rule_systemctl</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="s2">&#34;journalctl&#34;</span><span class="p">:</span> <span class="n">_rule_journalctl</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="s2">&#34;firewall-cmd&#34;</span><span class="p">:</span> <span class="n">_rule_firewall_cmd</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="s2">&#34;find&#34;</span><span class="p">:</span> <span class="n">_rule_find</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="s2">&#34;sqlite3&#34;</span><span class="p">:</span> <span class="n">_rule_sqlite3</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="s2">&#34;openssl&#34;</span><span class="p">:</span> <span class="n">_rule_openssl</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="s2">&#34;sshd&#34;</span><span class="p">:</span> <span class="n">_rule_sshd</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="s2">&#34;needs-restarting&#34;</span><span class="p">:</span> <span class="n">_rule_needs_restarting</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">    <span class="s2">&#34;dnf&#34;</span><span class="p">:</span> <span class="n">_rule_dnf</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="s2">&#34;rpm&#34;</span><span class="p">:</span> <span class="n">_rule_rpm</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="s2">&#34;ausearch&#34;</span><span class="p">:</span> <span class="n">_rule_ausearch</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="s2">&#34;grep&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">    <span class="s2">&#34;wc&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">    <span class="s2">&#34;stat&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="s2">&#34;du&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">    <span class="s2">&#34;df&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">    <span class="s2">&#34;test&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">    <span class="s2">&#34;id&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">    <span class="s2">&#34;uname&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">    <span class="s2">&#34;getenforce&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">    <span class="s2">&#34;sestatus&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="s2">&#34;getsebool&#34;</span><span class="p">:</span> <span class="n">_allow</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl"><span class="p">}</span></span></span></code></pre></div><p><code>READONLY_RULES</code> 是白名單本身：鍵是執行檔的檔名，值是那支工具的規則函式；不在這張表上的工具在 <code>readonly_violation</code> 裡被拒絕。其餘對到 <code>_allow</code> 的工具，任何參數都放行，因為它們沒有寫檔或執行其他程式的選項，判斷的依據在清單寫成 TOML 的那篇概念文章的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉白名單三層比對（執行檔的路徑、工具、工具的選項）裡的「工具」那一層。白名單要加一支工具時，在這張表加一列；那支工具有會寫檔或執行其他程式的選項時，同時寫一個規則函式，對到 <code>_allow</code> 等於宣告它的每一個選項都只讀。</p>
<h2 id="指令的只讀判定readonly_violation-與-checker-自己組的指令">指令的只讀判定：readonly_violation 與 Checker 自己組的指令</h2>
<p><code>readonly_violation</code> 把一條指令字串判定成放行或拒絕：先用 <code>shlex</code> 切詞，再依序做白名單的三層比對。執行檔的路徑那一層在這個函式裡比對；工具那一層依檔名查 <code>READONLY_RULES</code>，查不到就拒絕；工具的選項那一層交給查到的規則函式，寫在〈各工具的選項規則：systemctl、journalctl、firewall-cmd、ss、find、sqlite3、openssl、sshd、needs-restarting、dnf、rpm、ausearch〉。程式碼裡排在 <code>readonly_violation</code> 之前的，是腳本自己用清單的值組出來的指令（<code>LISTEN_COMMAND</code>、<code>timer_command</code>、<code>journal_command</code>），載入清單時與執行時用的是同一批函式，比對過的字串就是執行的字串。三層比對的理由，以及裸工具名依 <code>PATH</code> 找檔案而要經由 sudo 或 systemd 執行的條件，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># Checker 實際會拼出來的指令；validate 載入時用同一批函式先比對白名單，兩邊不會不一致。</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">LISTEN_COMMAND</span> <span class="o">=</span> <span class="s2">&#34;ss -H -l -n -t -u&#34;</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="n">UPDATE_TIMER_VERBS</span> <span class="o">=</span> <span class="p">((</span><span class="s2">&#34;is-enabled&#34;</span><span class="p">,</span> <span class="s2">&#34;enabled&#34;</span><span class="p">),</span> <span class="p">(</span><span class="s2">&#34;is-active&#34;</span><span class="p">,</span> <span class="s2">&#34;active&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="c1"># needs-restarting -r 判定要重開機時印的句子（dnf 4 與 dnf 5 相同，Runner 以 LC_ALL=C 執行）。</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="c1"># exit 1 也是 dnf 自己出錯時的 exit（ProtectSystem=strict 下寫不了 /var/log/dnf.log），只看 exit 分不開。</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="n">REBOOT_REQUIRED_TEXT</span> <span class="o">=</span> <span class="s2">&#34;Reboot is required&#34;</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">
</span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="k">def</span> <span class="nf">timer_command</span><span class="p">(</span><span class="n">verb</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">timer</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;systemctl </span><span class="si">{</span><span class="n">verb</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">shlex</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="n">timer</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">
</span></span><span class="line"><span class="ln">12</span><span class="cl">
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="k">def</span> <span class="nf">journal_command</span><span class="p">(</span><span class="n">unit</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;journalctl -u </span><span class="si">{</span><span class="n">shlex</span><span class="o">.</span><span class="n">quote</span><span class="p">(</span><span class="n">unit</span><span class="p">)</span><span class="si">}</span><span class="s2"> -n 1 -q --no-pager -o short-iso&#34;</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">
</span></span><span class="line"><span class="ln">16</span><span class="cl">
</span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="n">TRUSTED_BIN_DIRS</span> <span class="o">=</span> <span class="nb">frozenset</span><span class="p">({</span><span class="s2">&#34;/usr/bin&#34;</span><span class="p">,</span> <span class="s2">&#34;/usr/sbin&#34;</span><span class="p">,</span> <span class="s2">&#34;/bin&#34;</span><span class="p">,</span> <span class="s2">&#34;/sbin&#34;</span><span class="p">})</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">
</span></span><span class="line"><span class="ln">19</span><span class="cl">
</span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="k">def</span> <span class="nf">readonly_violation</span><span class="p">(</span><span class="n">command</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="s2">&#34;&#34;&#34;command 不在只讀白名單時回傳理由。&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="n">argv</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">    <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;指令切詞失敗：</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">argv</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34;指令是空的&#34;</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="k">if</span> <span class="s2">&#34;/&#34;</span> <span class="ow">in</span> <span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">and</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">TRUSTED_BIN_DIRS</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">}</span><span class="s2"> 的路徑不在 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">TRUSTED_BIN_DIRS</span><span class="p">))</span><span class="si">}</span><span class="s2"> 之下（只接受裸名或系統目錄）&#34;</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">    <span class="n">name</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">basename</span><span class="p">(</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">    <span class="n">rule</span> <span class="o">=</span> <span class="n">READONLY_RULES</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">name</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">    <span class="k">if</span> <span class="n">rule</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="s2"> 不在只讀白名單（</span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">READONLY_RULES</span><span class="p">))</span><span class="si">}</span><span class="s2">）&#34;</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">    <span class="k">return</span> <span class="n">rule</span><span class="p">([</span><span class="n">name</span><span class="p">]</span> <span class="o">+</span> <span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">:])</span></span></span></code></pre></div><p><code>timer_command</code> 與 <code>journal_command</code> 把清單裡的 <a href="/blog/linux/dotfile/knowledge-cards/systemd-timer/" data-link-title="systemd timer" data-link-desc="排程的工作沒有在預期的時間執行、主機關機期間錯過的排程不知道會不會補跑、要在 cron 與 systemd timer 之間選一種寫定期工作，或讀到 *.timer 檔與 systemctl list-timers 的輸出時">systemd timer</a> 名稱與 unit 名稱嵌進指令。<code>shlex.quote</code> 讓名稱裡的空白與引號在切詞之後仍是一個參數，名稱塞不進第二個參數；而名稱本身寫成選項的樣子時，<code>shlex.quote</code> 不改它，要靠白名單擋。單元測試 <code>test_evil_timer_rejected_at_load</code>（本篇沒有收錄）把清單的 timer 寫成 <code>--host=evil</code>，組出來的 <code>systemctl is-enabled --host=evil</code> 在載入時被 <code>systemctl</code> 的規則拒絕，腳本以 exit 2 結束，假的執行器一條指令都沒有收到。<code>REBOOT_REQUIRED_TEXT</code> 給自動更新那一項判斷要不要重開機，它跟 exit code 一起用的理由寫在註解裡，判定的程式在下面〈自動更新的排程、設定與重開機〉。註解寫這一句「dnf 4 與 dnf 5 相同」，原始碼與測試裡沒有 dnf 5 的輸出可以對照：單元測試用的是 dnf 4 格式的輸出，dnf 5 印的句子這次未實跑。</p>
<p>執行檔路徑那一層比對的是字串：<code>os.path.dirname</code> 取出的目錄要逐字等於四個系統目錄之一，沒有先正規化路徑。<code>./grep</code> 的目錄是 <code>.</code>、<code>/usr/local/bin/grep</code> 的目錄是 <code>/usr/local/bin</code>，都不在表上；<code>/usr/bin/../../tmp/grep</code> 的目錄是 <code>/usr/bin/../../tmp</code>，同樣不等於 <code>/usr/bin</code>，<code>..</code> 因此繞不過這一層。規則函式收到的第一個元素換成了檔名，<code>/usr/bin/grep</code> 與 <code>grep</code> 走同一個規則。<code>shlex.split</code> 遇到沒有收尾的引號時丟出 <code>ValueError</code>，這一條指令判成拒絕，不讓整支腳本中止。</p>
<p>白名單是擋得住「請 PID 1 這類別的程序代做」這種改動的那一層。systemd 沙箱的檔案系統與網路限制管的是腳本自己的程序，而 <code>systemctl stop</code> 經由 D-Bus 請 systemd（PID 1）去停服務，停服務的是不受這些限制的 systemd。在 Ubuntu 24.04（systemd 255）上實跑：<code>systemd-run --wait --pipe -p ProtectSystem=strict -p IPAddressDeny=any systemctl stop &lt;unit&gt;</code> 回 rc=0，unit 被停掉，加上 <code>PrivateNetwork=yes</code> 也一樣停得掉；只有用 <code>RestrictAddressFamilies=&quot;AF_INET AF_INET6&quot;</code> 拿掉 AF_UNIX 時，回 rc=1（<code>Failed to connect to bus: Address family not supported by protocol</code>）。所以 <code>systemctl</code> 的規則只放行查詢的子命令；白名單與沙箱怎麼分工，在清單寫成 TOML 的那篇概念文章的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</p>
<h2 id="清單的載入與驗證probe_validatorload_manifest-與-validate">清單的載入與驗證：Probe、_Validator、load_manifest 與 validate</h2>
<p>這一段把 TOML 讀成 Python 的 dict 之後逐段驗證並正規化：每個欄位查存在、型別與允許值，字串裡的 <code>{{名稱}}</code> 換成 <code>[vars]</code> 的值，要執行的指令比對白名單；整份清單的錯誤收齊之後一次丟出 <code>ManifestError</code>，這時一條指令都還沒有執行。欄位怎麼依檢查腳本與人的需要設計，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈清單欄位的設計：檢查腳本要讀的與人要讀的〉；照這段驗證實際要求的欄位寫成範本，在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈變數檔與範本：檢查腳本實際要求的欄位、兼任兩種角色的主機〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">  1</span><span class="cl"><span class="c1"># ── 清單解析與驗證 ─────────────────────────────────────────</span>
</span></span><span class="line"><span class="ln">  2</span><span class="cl">
</span></span><span class="line"><span class="ln">  3</span><span class="cl">
</span></span><span class="line"><span class="ln">  4</span><span class="cl"><span class="nd">@dataclass</span>
</span></span><span class="line"><span class="ln">  5</span><span class="cl"><span class="k">class</span> <span class="nc">Probe</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">  6</span><span class="cl">    <span class="n">command</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="ln">  7</span><span class="cl">    <span class="n">expect_exit</span><span class="p">:</span> <span class="nb">int</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">  8</span><span class="cl">    <span class="n">ok_exit</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">int</span><span class="p">]</span> <span class="o">=</span> <span class="n">field</span><span class="p">(</span><span class="n">default_factory</span><span class="o">=</span><span class="k">lambda</span><span class="p">:</span> <span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">  9</span><span class="cl">    <span class="n">expect_stdout</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 10</span><span class="cl">    <span class="n">expect_contains</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 11</span><span class="cl">    <span class="n">expect_not_contains</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 12</span><span class="cl">    <span class="n">measure</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 13</span><span class="cl">    <span class="n">warn_above</span><span class="p">:</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 14</span><span class="cl">
</span></span><span class="line"><span class="ln"> 15</span><span class="cl">
</span></span><span class="line"><span class="ln"> 16</span><span class="cl"><span class="k">class</span> <span class="nc">_Validator</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 17</span><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">variables</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]):</span>
</span></span><span class="line"><span class="ln"> 18</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">errors</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 19</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">vars</span> <span class="o">=</span> <span class="n">variables</span>
</span></span><span class="line"><span class="ln"> 20</span><span class="cl">
</span></span><span class="line"><span class="ln"> 21</span><span class="cl">    <span class="k">def</span> <span class="nf">err</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">msg</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 22</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">where</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 23</span><span class="cl">
</span></span><span class="line"><span class="ln"> 24</span><span class="cl">    <span class="k">def</span> <span class="nf">subst</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 25</span><span class="cl">        <span class="k">def</span> <span class="nf">repl</span><span class="p">(</span><span class="n">m</span><span class="p">:</span> <span class="n">re</span><span class="o">.</span><span class="n">Match</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 26</span><span class="cl">            <span class="n">name</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 27</span><span class="cl">            <span class="k">if</span> <span class="n">name</span> <span class="ow">not</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">vars</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 28</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="se">{{{{</span><span class="si">{</span><span class="n">name</span><span class="si">}</span><span class="se">}}}}</span><span class="s2"> 沒有定義在 [vars]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 29</span><span class="cl">                <span class="k">return</span> <span class="n">m</span><span class="o">.</span><span class="n">group</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 30</span><span class="cl">            <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">vars</span><span class="p">[</span><span class="n">name</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 31</span><span class="cl">
</span></span><span class="line"><span class="ln"> 32</span><span class="cl">        <span class="k">return</span> <span class="n">VAR_RE</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="n">repl</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 33</span><span class="cl">
</span></span><span class="line"><span class="ln"> 34</span><span class="cl">    <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tbl</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">typ</span><span class="p">:</span> <span class="nb">type</span> <span class="o">|</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">type</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">required</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">,</span> <span class="n">enum</span><span class="p">:</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="o">...</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Any</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 35</span><span class="cl">        <span class="k">if</span> <span class="n">key</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">tbl</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 36</span><span class="cl">            <span class="k">if</span> <span class="n">required</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 37</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;缺少必填欄位 </span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 38</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 39</span><span class="cl">        <span class="n">v</span> <span class="o">=</span> <span class="n">tbl</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 40</span><span class="cl">        <span class="c1"># bool 是 int 的子類別，數字欄位不收 true/false。</span>
</span></span><span class="line"><span class="ln"> 41</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">typ</span><span class="p">)</span> <span class="ow">or</span> <span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">bool</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">bool</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="n">typ</span> <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">typ</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">)</span> <span class="k">else</span> <span class="p">(</span><span class="n">typ</span><span class="p">,))):</span>
</span></span><span class="line"><span class="ln"> 42</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> 的型別要是 </span><span class="si">{</span><span class="n">_typename</span><span class="p">(</span><span class="n">typ</span><span class="p">)</span><span class="si">}</span><span class="s2">，實際是 </span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="n">v</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 43</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 44</span><span class="cl">        <span class="k">if</span> <span class="n">enum</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">v</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">enum</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 45</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> 要是 </span><span class="si">{</span><span class="nb">list</span><span class="p">(</span><span class="n">enum</span><span class="p">)</span><span class="si">}</span><span class="s2"> 之一，實際是 </span><span class="si">{</span><span class="n">v</span><span class="si">!r}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 46</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 47</span><span class="cl">        <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 48</span><span class="cl">            <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">subst</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">where</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 49</span><span class="cl">        <span class="k">return</span> <span class="n">v</span>
</span></span><span class="line"><span class="ln"> 50</span><span class="cl">
</span></span><span class="line"><span class="ln"> 51</span><span class="cl">    <span class="k">def</span> <span class="nf">reject_old_command_name</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tbl</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 52</span><span class="cl">        <span class="c1"># command 一律會被執行；只印不執行的欄位叫 manual_command，舊名出現在只印的位置判成格式錯。</span>
</span></span><span class="line"><span class="ln"> 53</span><span class="cl">        <span class="k">if</span> <span class="s2">&#34;command&#34;</span> <span class="ow">in</span> <span class="n">tbl</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 54</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="s2">&#34;只印不執行的指令改用 manual_command（command 一律會被執行並過白名單）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 55</span><span class="cl">
</span></span><span class="line"><span class="ln"> 56</span><span class="cl">    <span class="k">def</span> <span class="nf">str_list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tbl</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">required</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 57</span><span class="cl">        <span class="n">v</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="nb">list</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 58</span><span class="cl">        <span class="k">if</span> <span class="n">v</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 59</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 60</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">v</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 61</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> 的每一項都要是字串&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 62</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 63</span><span class="cl">        <span class="k">return</span> <span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">subst</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">where</span><span class="si">}</span><span class="s2">.</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">x</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">v</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 64</span><span class="cl">
</span></span><span class="line"><span class="ln"> 65</span><span class="cl">    <span class="k">def</span> <span class="nf">table_list</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">data</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">key</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">required</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">True</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]]:</span>
</span></span><span class="line"><span class="ln"> 66</span><span class="cl">        <span class="n">v</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 67</span><span class="cl">        <span class="k">if</span> <span class="n">v</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 68</span><span class="cl">            <span class="k">if</span> <span class="n">required</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 69</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;缺少 [[</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">]]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 70</span><span class="cl">            <span class="k">return</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 71</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">v</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 72</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2"> 要是 [[</span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">]] 表格陣列&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 73</span><span class="cl">            <span class="k">return</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 74</span><span class="cl">        <span class="k">return</span> <span class="n">v</span>
</span></span><span class="line"><span class="ln"> 75</span><span class="cl">
</span></span><span class="line"><span class="ln"> 76</span><span class="cl">    <span class="k">def</span> <span class="nf">probe</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tbl</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">allow_measure</span><span class="p">:</span> <span class="nb">bool</span> <span class="o">=</span> <span class="kc">False</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Probe</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 77</span><span class="cl">        <span class="n">command</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;command&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">where</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 78</span><span class="cl">        <span class="k">if</span> <span class="n">command</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 79</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 80</span><span class="cl">        <span class="n">reason</span> <span class="o">=</span> <span class="n">readonly_violation</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 81</span><span class="cl">        <span class="k">if</span> <span class="n">reason</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 82</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;command 不是只讀指令：</span><span class="si">{</span><span class="n">reason</span><span class="si">}</span><span class="s2">（</span><span class="si">{</span><span class="n">command</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 83</span><span class="cl">        <span class="n">p</span> <span class="o">=</span> <span class="n">Probe</span><span class="p">(</span><span class="n">command</span><span class="o">=</span><span class="n">command</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 84</span><span class="cl">        <span class="n">p</span><span class="o">.</span><span class="n">expect_exit</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;expect_exit&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 85</span><span class="cl">        <span class="n">ok_exit</span> <span class="o">=</span> <span class="n">tbl</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;ok_exit&#34;</span><span class="p">,</span> <span class="p">[</span><span class="mi">0</span><span class="p">])</span>
</span></span><span class="line"><span class="ln"> 86</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">ok_exit</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">int</span><span class="p">)</span> <span class="ow">and</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">bool</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">ok_exit</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 87</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="s2">&#34;ok_exit 要是整數陣列&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 88</span><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 89</span><span class="cl">            <span class="n">p</span><span class="o">.</span><span class="n">ok_exit</span> <span class="o">=</span> <span class="n">ok_exit</span>
</span></span><span class="line"><span class="ln"> 90</span><span class="cl">        <span class="n">p</span><span class="o">.</span><span class="n">expect_stdout</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;expect_stdout&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 91</span><span class="cl">        <span class="n">p</span><span class="o">.</span><span class="n">expect_contains</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;expect_contains&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 92</span><span class="cl">        <span class="n">p</span><span class="o">.</span><span class="n">expect_not_contains</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;expect_not_contains&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 93</span><span class="cl">        <span class="k">if</span> <span class="n">allow_measure</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 94</span><span class="cl">            <span class="n">p</span><span class="o">.</span><span class="n">measure</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;measure&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">MEASURE_VALUES</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 95</span><span class="cl">            <span class="n">p</span><span class="o">.</span><span class="n">warn_above</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">tbl</span><span class="p">,</span> <span class="s2">&#34;warn_above&#34;</span><span class="p">,</span> <span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">),</span> <span class="n">where</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="n">p</span><span class="o">.</span><span class="n">measure</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 96</span><span class="cl">        <span class="n">has_expect</span> <span class="o">=</span> <span class="nb">any</span><span class="p">(</span><span class="n">x</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">expect_exit</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_stdout</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_contains</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_not_contains</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">measure</span><span class="p">))</span>
</span></span><span class="line"><span class="ln"> 97</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">has_expect</span> <span class="ow">and</span> <span class="s2">&#34;ok_exit&#34;</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">tbl</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 98</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="s2">&#34;探針至少要有一個預期值（expect_exit、ok_exit、expect_stdout、expect_contains、expect_not_contains 或 measure）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 99</span><span class="cl">        <span class="k">return</span> <span class="n">p</span>
</span></span><span class="line"><span class="ln">100</span><span class="cl">
</span></span><span class="line"><span class="ln">101</span><span class="cl">
</span></span><span class="line"><span class="ln">102</span><span class="cl"><span class="k">def</span> <span class="nf">_typename</span><span class="p">(</span><span class="n">typ</span><span class="p">:</span> <span class="nb">type</span> <span class="o">|</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">type</span><span class="p">,</span> <span class="o">...</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">103</span><span class="cl">    <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">typ</span><span class="p">,</span> <span class="nb">tuple</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">104</span><span class="cl">        <span class="k">return</span> <span class="s2">&#34; 或 &#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">t</span><span class="o">.</span><span class="vm">__name__</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">typ</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">105</span><span class="cl">    <span class="k">return</span> <span class="n">typ</span><span class="o">.</span><span class="vm">__name__</span></span></span></code></pre></div><p><code>Probe</code> 是一個探針：要執行的一條指令，加上怎樣算正常的預期值。<code>_Validator</code> 把驗證時共用的動作收在一起，每個方法查到錯誤時記一條、回傳 <code>None</code> 或空的結果（<code>table_list</code> 回空串列），不丟例外，所以一個欄位錯了之後，其他欄位照樣往下驗。</p>
<p><code>get</code> 的型別比對多了一個布林的判斷：Python 的 <code>bool</code> 是 <code>int</code> 的子類別，只用 <code>isinstance(v, int)</code> 的話，<code>port = true</code> 會被當成整數 1 收下。<code>get</code> 先比對型別與允許值，再換掉字串裡的 <code>{{名稱}}</code>：有允許值的欄位（<code>exposure</code>、<code>where</code>、<code>protocol</code> 這類）寫成 <code>{{名稱}}</code> 時，比對的是 <code>{{名稱}}</code> 這串字本身，判成格式錯，所以這些欄位不能用 <code>[vars]</code> 填。之後比對白名單的是換過的字串，所以命令列 <code>--set</code> 傳進來的值跟清單裡寫死的值走同一道白名單。<code>[vars]</code> 沒有的名稱記一條錯誤，原文的 <code>{{名稱}}</code> 留在字串裡。</p>
<p><code>reject_old_command_name</code> 管的是欄位名稱的意思：清單裡的 <code>command</code> 一律會被執行，只印給人看的指令寫在 <code>manual_command</code>。只印的位置（<code>[[log]]</code>、暫停與恢復的步驟、主機外的確認）出現 <code>command</code> 時判成格式錯，讀清單的人看到 <code>command</code> 就知道那一條會被執行，單元測試 <code>test_log_old_name_rejected</code>、<code>test_resume_old_name_rejected</code> 與 <code>test_external_verify_old_name_rejected</code>（本篇沒有收錄）各測一個位置。</p>
<p><code>probe</code> 要求探針至少寫出一個預期值，或明寫 <code>ok_exit</code>：<code>ok_exit</code> 的預設 <code>[0]</code> 不算宣告，清單的讀者要看得到這一項怎樣算正常。<code>measure</code> 與 <code>warn_above</code> 只在 <code>allow_measure</code> 為真時讀，也就是 log 的計數門檻（<code>[[log.alert]]</code>）；寫了 <code>measure</code> 就一定要有 <code>warn_above</code>，量到的值才有門檻可比。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">  1</span><span class="cl"><span class="k">def</span> <span class="nf">load_manifest</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">overrides</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">  2</span><span class="cl">    <span class="s2">&#34;&#34;&#34;讀取並驗證清單，回傳正規化後的結構；格式錯時丟 ManifestError。&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">  3</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">  4</span><span class="cl">        <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="s2">&#34;rb&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">  5</span><span class="cl">            <span class="n">data</span> <span class="o">=</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">  6</span><span class="cl">    <span class="k">except</span> <span class="ne">OSError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">  7</span><span class="cl">        <span class="k">raise</span> <span class="n">ManifestError</span><span class="p">([</span><span class="sa">f</span><span class="s2">&#34;讀不到清單 </span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2">：</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">])</span> <span class="kn">from</span> <span class="nn">e</span>
</span></span><span class="line"><span class="ln">  8</span><span class="cl">    <span class="k">except</span> <span class="n">tomllib</span><span class="o">.</span><span class="n">TOMLDecodeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">  9</span><span class="cl">        <span class="k">raise</span> <span class="n">ManifestError</span><span class="p">([</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2"> 不是合法的 TOML 1.0：</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">])</span> <span class="kn">from</span> <span class="nn">e</span>
</span></span><span class="line"><span class="ln"> 10</span><span class="cl">    <span class="k">return</span> <span class="n">validate</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">overrides</span> <span class="ow">or</span> <span class="p">{})</span>
</span></span><span class="line"><span class="ln"> 11</span><span class="cl">
</span></span><span class="line"><span class="ln"> 12</span><span class="cl">
</span></span><span class="line"><span class="ln"> 13</span><span class="cl"><span class="k">def</span> <span class="nf">validate</span><span class="p">(</span><span class="n">data</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">overrides</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 14</span><span class="cl">    <span class="n">raw_vars</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;vars&#34;</span><span class="p">,</span> <span class="p">{})</span>
</span></span><span class="line"><span class="ln"> 15</span><span class="cl">    <span class="n">errors</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 16</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">raw_vars</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">raw_vars</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
</span></span><span class="line"><span class="ln"> 17</span><span class="cl">        <span class="n">errors</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="s2">&#34;vars: [vars] 的每個值都要是字串&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 18</span><span class="cl">        <span class="n">raw_vars</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln"> 19</span><span class="cl">    <span class="n">variables</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">raw_vars</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 20</span><span class="cl">    <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="n">overrides</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 21</span><span class="cl">        <span class="k">if</span> <span class="n">k</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">variables</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 22</span><span class="cl">            <span class="n">errors</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;--set </span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2">: [vars] 沒有這個名稱&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 23</span><span class="cl">    <span class="n">variables</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">overrides</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 24</span><span class="cl">    <span class="n">v</span> <span class="o">=</span> <span class="n">_Validator</span><span class="p">(</span><span class="n">variables</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 25</span><span class="cl">    <span class="n">v</span><span class="o">.</span><span class="n">errors</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">errors</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 26</span><span class="cl">    <span class="n">out</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;vars&#34;</span><span class="p">:</span> <span class="n">variables</span><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 27</span><span class="cl">
</span></span><span class="line"><span class="ln"> 28</span><span class="cl">    <span class="n">host</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;host&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 29</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 30</span><span class="cl">        <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;host&#34;</span><span class="p">,</span> <span class="s2">&#34;缺少 [host]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 31</span><span class="cl">        <span class="n">host</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln"> 32</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;host&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">host</span><span class="p">,</span> <span class="n">k</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="s2">&#34;host&#34;</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;name&#34;</span><span class="p">,</span> <span class="s2">&#34;provider&#34;</span><span class="p">,</span> <span class="s2">&#34;region&#34;</span><span class="p">,</span> <span class="s2">&#34;role&#34;</span><span class="p">)}</span>
</span></span><span class="line"><span class="ln"> 33</span><span class="cl">
</span></span><span class="line"><span class="ln"> 34</span><span class="cl">    <span class="n">la</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">,</span> <span class="p">{</span><span class="s2">&#34;protocols&#34;</span><span class="p">:</span> <span class="p">[</span><span class="s2">&#34;tcp&#34;</span><span class="p">],</span> <span class="s2">&#34;allow&#34;</span><span class="p">:</span> <span class="p">[]})</span>
</span></span><span class="line"><span class="ln"> 35</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">la</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 36</span><span class="cl">        <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">,</span> <span class="s2">&#34;要是表格&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 37</span><span class="cl">        <span class="n">la</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln"> 38</span><span class="cl">    <span class="n">protocols</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">str_list</span><span class="p">(</span><span class="n">la</span><span class="p">,</span> <span class="s2">&#34;protocols&#34;</span><span class="p">,</span> <span class="s2">&#34;listen_audit&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 39</span><span class="cl">    <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">protocols</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 40</span><span class="cl">        <span class="k">if</span> <span class="n">p</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">PROTOCOL_VALUES</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 41</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;protocols 只能是 </span><span class="si">{</span><span class="nb">list</span><span class="p">(</span><span class="n">PROTOCOL_VALUES</span><span class="p">)</span><span class="si">}</span><span class="s2">，實際有 </span><span class="si">{</span><span class="n">p</span><span class="si">!r}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 42</span><span class="cl">    <span class="n">allow_out</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 43</span><span class="cl">    <span class="n">allow</span> <span class="o">=</span> <span class="n">la</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;allow&#34;</span><span class="p">,</span> <span class="p">[])</span>
</span></span><span class="line"><span class="ln"> 44</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">allow</span><span class="p">,</span> <span class="nb">list</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 45</span><span class="cl">        <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">,</span> <span class="s2">&#34;allow 要是陣列&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 46</span><span class="cl">        <span class="n">allow</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 47</span><span class="cl">    <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">a</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">allow</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 48</span><span class="cl">        <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;listen_audit.allow[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln"> 49</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 50</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;要是 { port = ..., protocol = ..., reason = ... }&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 51</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln"> 52</span><span class="cl">        <span class="n">allow_out</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s2">&#34;port&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="s2">&#34;port&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span> <span class="s2">&#34;protocol&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="s2">&#34;protocol&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">PROTOCOL_VALUES</span><span class="p">),</span> <span class="s2">&#34;reason&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="s2">&#34;reason&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">)})</span>
</span></span><span class="line"><span class="ln"> 53</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;protocols&#34;</span><span class="p">:</span> <span class="n">protocols</span><span class="p">,</span> <span class="s2">&#34;allow&#34;</span><span class="p">:</span> <span class="n">allow_out</span><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 54</span><span class="cl">
</span></span><span class="line"><span class="ln"> 55</span><span class="cl">    <span class="n">eps</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 56</span><span class="cl">    <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">ep</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">table_list</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="s2">&#34;entrypoint&#34;</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln"> 57</span><span class="cl">        <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;entrypoint[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln"> 58</span><span class="cl">        <span class="n">name</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ep</span><span class="p">,</span> <span class="s2">&#34;name&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 59</span><span class="cl">        <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;entrypoint[</span><span class="si">{</span><span class="n">name</span> <span class="ow">or</span> <span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln"> 60</span><span class="cl">        <span class="n">e</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln"> 61</span><span class="cl">            <span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
</span></span><span class="line"><span class="ln"> 62</span><span class="cl">            <span class="s2">&#34;port&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ep</span><span class="p">,</span> <span class="s2">&#34;port&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln"> 63</span><span class="cl">            <span class="s2">&#34;protocol&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ep</span><span class="p">,</span> <span class="s2">&#34;protocol&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">PROTOCOL_VALUES</span><span class="p">),</span>
</span></span><span class="line"><span class="ln"> 64</span><span class="cl">            <span class="s2">&#34;exposure&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ep</span><span class="p">,</span> <span class="s2">&#34;exposure&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">EXPOSURE_VALUES</span><span class="p">),</span>
</span></span><span class="line"><span class="ln"> 65</span><span class="cl">            <span class="s2">&#34;where&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ep</span><span class="p">,</span> <span class="s2">&#34;where&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">WHERE_VALUES</span><span class="p">),</span>
</span></span><span class="line"><span class="ln"> 66</span><span class="cl">            <span class="s2">&#34;enforced_by&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">str_list</span><span class="p">(</span><span class="n">ep</span><span class="p">,</span> <span class="s2">&#34;enforced_by&#34;</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[],</span>
</span></span><span class="line"><span class="ln"> 67</span><span class="cl">            <span class="s2">&#34;verify&#34;</span><span class="p">:</span> <span class="p">[],</span>
</span></span><span class="line"><span class="ln"> 68</span><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="ln"> 69</span><span class="cl">        <span class="n">verifies</span> <span class="o">=</span> <span class="n">ep</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;verify&#34;</span><span class="p">,</span> <span class="p">[])</span>
</span></span><span class="line"><span class="ln"> 70</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">verifies</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">verifies</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 71</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;verify 要是 [[entrypoint.verify]] 表格陣列&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 72</span><span class="cl">            <span class="n">verifies</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 73</span><span class="cl">        <span class="n">layers_seen</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 74</span><span class="cl">        <span class="k">for</span> <span class="n">j</span><span class="p">,</span> <span class="n">vf</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">verifies</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 75</span><span class="cl">            <span class="n">vw</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">w</span><span class="si">}</span><span class="s2">.verify[</span><span class="si">{</span><span class="n">j</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln"> 76</span><span class="cl">            <span class="n">layer</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="s2">&#34;layer&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">vw</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 77</span><span class="cl">            <span class="n">where</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="s2">&#34;where&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">vw</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">WHERE_VALUES</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 78</span><span class="cl">            <span class="n">layers_seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">layer</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 79</span><span class="cl">            <span class="n">item</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;layer&#34;</span><span class="p">:</span> <span class="n">layer</span><span class="p">,</span> <span class="s2">&#34;where&#34;</span><span class="p">:</span> <span class="n">where</span><span class="p">}</span>
</span></span><span class="line"><span class="ln"> 80</span><span class="cl">            <span class="k">if</span> <span class="n">where</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 81</span><span class="cl">                <span class="n">item</span><span class="p">[</span><span class="s2">&#34;probe&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">probe</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="n">vw</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 82</span><span class="cl">            <span class="k">elif</span> <span class="n">where</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 83</span><span class="cl">                <span class="n">item</span><span class="p">[</span><span class="s2">&#34;console_path&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="s2">&#34;console_path&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">vw</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 84</span><span class="cl">                <span class="n">v</span><span class="o">.</span><span class="n">reject_old_command_name</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="n">vw</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 85</span><span class="cl">                <span class="n">item</span><span class="p">[</span><span class="s2">&#34;manual_command&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="s2">&#34;manual_command&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">vw</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 86</span><span class="cl">                <span class="k">if</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;console_path&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;manual_command&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 87</span><span class="cl">                    <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">vw</span><span class="p">,</span> <span class="s2">&#34;主機外的確認要有 console_path 或 manual_command&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 88</span><span class="cl">                <span class="n">item</span><span class="p">[</span><span class="s2">&#34;expect&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="s2">&#34;expect&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">vw</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 89</span><span class="cl">            <span class="n">e</span><span class="p">[</span><span class="s2">&#34;verify&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 90</span><span class="cl">        <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;enforced_by&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 91</span><span class="cl">            <span class="k">if</span> <span class="n">layer</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">layers_seen</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 92</span><span class="cl">                <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;enforced_by 的 </span><span class="si">{</span><span class="n">layer</span><span class="si">!r}</span><span class="s2"> 沒有對應的 [[entrypoint.verify]]（layer = </span><span class="si">{</span><span class="n">layer</span><span class="si">!r}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 93</span><span class="cl">        <span class="k">for</span> <span class="n">layer</span> <span class="ow">in</span> <span class="n">layers_seen</span> <span class="o">-</span> <span class="nb">set</span><span class="p">(</span><span class="n">e</span><span class="p">[</span><span class="s2">&#34;enforced_by&#34;</span><span class="p">])</span> <span class="o">-</span> <span class="p">{</span><span class="kc">None</span><span class="p">}:</span>
</span></span><span class="line"><span class="ln"> 94</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;[[entrypoint.verify]] 的 layer </span><span class="si">{</span><span class="n">layer</span><span class="si">!r}</span><span class="s2"> 不在 enforced_by 裡&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 95</span><span class="cl">        <span class="n">eps</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">e</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 96</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;entrypoint&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">eps</span>
</span></span><span class="line"><span class="ln"> 97</span><span class="cl">
</span></span><span class="line"><span class="ln"> 98</span><span class="cl">    <span class="n">logs</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 99</span><span class="cl">    <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">lg</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">table_list</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="s2">&#34;log&#34;</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">100</span><span class="cl">        <span class="n">name</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">lg</span><span class="p">,</span> <span class="s2">&#34;name&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;log[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">101</span><span class="cl">        <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;log[</span><span class="si">{</span><span class="n">name</span> <span class="ow">or</span> <span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln">102</span><span class="cl">        <span class="n">v</span><span class="o">.</span><span class="n">reject_old_command_name</span><span class="p">(</span><span class="n">lg</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">103</span><span class="cl">        <span class="n">source</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">lg</span><span class="p">,</span> <span class="s2">&#34;source&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">104</span><span class="cl">        <span class="k">if</span> <span class="n">source</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">re</span><span class="o">.</span><span class="k">match</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;^(file|journal):.+&#34;</span><span class="p">,</span> <span class="n">source</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">105</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;source 要寫成 file:&lt;路徑&gt; 或 journal:&lt;unit&gt;，實際是 </span><span class="si">{</span><span class="n">source</span><span class="si">!r}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">106</span><span class="cl">        <span class="n">item</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln">107</span><span class="cl">            <span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">108</span><span class="cl">            <span class="s2">&#34;source&#34;</span><span class="p">:</span> <span class="n">source</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">109</span><span class="cl">            <span class="s2">&#34;look_for&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">lg</span><span class="p">,</span> <span class="s2">&#34;look_for&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">110</span><span class="cl">            <span class="s2">&#34;interval_days&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">lg</span><span class="p">,</span> <span class="s2">&#34;interval_days&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">111</span><span class="cl">            <span class="s2">&#34;manual_command&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">lg</span><span class="p">,</span> <span class="s2">&#34;manual_command&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">112</span><span class="cl">            <span class="s2">&#34;alerts&#34;</span><span class="p">:</span> <span class="p">[],</span>
</span></span><span class="line"><span class="ln">113</span><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="ln">114</span><span class="cl">        <span class="n">alerts</span> <span class="o">=</span> <span class="n">lg</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;alert&#34;</span><span class="p">,</span> <span class="p">[])</span>
</span></span><span class="line"><span class="ln">115</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">alerts</span><span class="p">,</span> <span class="nb">list</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">alerts</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">116</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;alert 要是 [[log.alert]] 表格陣列&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">117</span><span class="cl">            <span class="n">alerts</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">118</span><span class="cl">        <span class="k">for</span> <span class="n">j</span><span class="p">,</span> <span class="n">al</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">alerts</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">119</span><span class="cl">            <span class="n">aw</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">w</span><span class="si">}</span><span class="s2">.alert[</span><span class="si">{</span><span class="n">j</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln">120</span><span class="cl">            <span class="n">aname</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">al</span><span class="p">,</span> <span class="s2">&#34;name&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">aw</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">121</span><span class="cl">            <span class="n">item</span><span class="p">[</span><span class="s2">&#34;alerts&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">aname</span><span class="p">,</span> <span class="s2">&#34;probe&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">probe</span><span class="p">(</span><span class="n">al</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">w</span><span class="si">}</span><span class="s2">.alert[</span><span class="si">{</span><span class="n">aname</span> <span class="ow">or</span> <span class="n">j</span><span class="si">}</span><span class="s2">]&#34;</span><span class="p">,</span> <span class="n">allow_measure</span><span class="o">=</span><span class="kc">True</span><span class="p">)})</span>
</span></span><span class="line"><span class="ln">122</span><span class="cl">        <span class="n">logs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">123</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;log&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">logs</span>
</span></span><span class="line"><span class="ln">124</span><span class="cl">
</span></span><span class="line"><span class="ln">125</span><span class="cl">    <span class="n">accs</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">126</span><span class="cl">    <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">ac</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">table_list</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="s2">&#34;accumulation&#34;</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">127</span><span class="cl">        <span class="n">name</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;name&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;accumulation[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">128</span><span class="cl">        <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;accumulation[</span><span class="si">{</span><span class="n">name</span> <span class="ow">or</span> <span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln">129</span><span class="cl">        <span class="n">paths</span> <span class="o">=</span> <span class="n">ac</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;path&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">130</span><span class="cl">        <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">paths</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">131</span><span class="cl">            <span class="n">paths</span> <span class="o">=</span> <span class="p">[</span><span class="n">paths</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">132</span><span class="cl">            <span class="n">ac</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="n">path</span><span class="o">=</span><span class="n">paths</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">133</span><span class="cl">        <span class="n">item</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln">134</span><span class="cl">            <span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">135</span><span class="cl">            <span class="s2">&#34;paths&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">str_list</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;path&#34;</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[],</span>
</span></span><span class="line"><span class="ln">136</span><span class="cl">            <span class="s2">&#34;missing_ok&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;missing_ok&#34;</span><span class="p">,</span> <span class="nb">bool</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="ow">or</span> <span class="kc">False</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">137</span><span class="cl">            <span class="s2">&#34;limit_bytes&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;limit_bytes&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">138</span><span class="cl">            <span class="s2">&#34;retention_days&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;retention_days&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">139</span><span class="cl">            <span class="s2">&#34;purge_command&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;purge_command&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">140</span><span class="cl">            <span class="s2">&#34;oldest_command&#34;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">141</span><span class="cl">        <span class="p">}</span>
</span></span><span class="line"><span class="ln">142</span><span class="cl">        <span class="k">if</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;limit_bytes&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;retention_days&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">143</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;limit_bytes 與 retention_days 至少要有一個&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">144</span><span class="cl">        <span class="n">oc</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">ac</span><span class="p">,</span> <span class="s2">&#34;oldest_command&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">145</span><span class="cl">        <span class="k">if</span> <span class="n">oc</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">146</span><span class="cl">            <span class="k">if</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;retention_days&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">147</span><span class="cl">                <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;有 oldest_command 時要有 retention_days&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">148</span><span class="cl">            <span class="n">reason</span> <span class="o">=</span> <span class="n">readonly_violation</span><span class="p">(</span><span class="n">oc</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">149</span><span class="cl">            <span class="k">if</span> <span class="n">reason</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">150</span><span class="cl">                <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;oldest_command 不是只讀指令：</span><span class="si">{</span><span class="n">reason</span><span class="si">}</span><span class="s2">（</span><span class="si">{</span><span class="n">oc</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">151</span><span class="cl">            <span class="n">item</span><span class="p">[</span><span class="s2">&#34;oldest_command&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">oc</span>
</span></span><span class="line"><span class="ln">152</span><span class="cl">        <span class="n">accs</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">153</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;accumulation&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">accs</span>
</span></span><span class="line"><span class="ln">154</span><span class="cl">
</span></span><span class="line"><span class="ln">155</span><span class="cl">    <span class="n">fss</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">156</span><span class="cl">    <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">fs</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">table_list</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="s2">&#34;filesystem&#34;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">157</span><span class="cl">        <span class="n">name</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">fs</span><span class="p">,</span> <span class="s2">&#34;name&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;filesystem[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">158</span><span class="cl">        <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;filesystem[</span><span class="si">{</span><span class="n">name</span> <span class="ow">or</span> <span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln">159</span><span class="cl">        <span class="n">fss</span><span class="o">.</span><span class="n">append</span><span class="p">({</span><span class="s2">&#34;name&#34;</span><span class="p">:</span> <span class="n">name</span><span class="p">,</span> <span class="s2">&#34;mount&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">fs</span><span class="p">,</span> <span class="s2">&#34;mount&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span> <span class="s2">&#34;min_free_bytes&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">fs</span><span class="p">,</span> <span class="s2">&#34;min_free_bytes&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">)})</span>
</span></span><span class="line"><span class="ln">160</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;filesystem&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">fss</span>
</span></span><span class="line"><span class="ln">161</span><span class="cl">
</span></span><span class="line"><span class="ln">162</span><span class="cl">    <span class="n">up</span> <span class="o">=</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;updates&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">163</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">up</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">164</span><span class="cl">        <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;updates&#34;</span><span class="p">,</span> <span class="s2">&#34;缺少 [updates]&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">165</span><span class="cl">        <span class="n">up</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln">166</span><span class="cl">    <span class="n">expect</span> <span class="o">=</span> <span class="n">up</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;expect&#34;</span><span class="p">,</span> <span class="p">{})</span>
</span></span><span class="line"><span class="ln">167</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">expect</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">all</span><span class="p">(</span><span class="nb">isinstance</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">expect</span><span class="o">.</span><span class="n">values</span><span class="p">()):</span>
</span></span><span class="line"><span class="ln">168</span><span class="cl">        <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;updates&#34;</span><span class="p">,</span> <span class="s2">&#34;expect 要是值為字串的 inline table&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">169</span><span class="cl">        <span class="n">expect</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln">170</span><span class="cl">    <span class="n">reboot</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">up</span><span class="p">,</span> <span class="s2">&#34;reboot_check&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="s2">&#34;updates&#34;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">171</span><span class="cl">    <span class="k">if</span> <span class="n">reboot</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">172</span><span class="cl">        <span class="n">reason</span> <span class="o">=</span> <span class="n">readonly_violation</span><span class="p">(</span><span class="n">reboot</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">173</span><span class="cl">        <span class="k">if</span> <span class="n">reason</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">174</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="s2">&#34;updates&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;reboot_check 不是只讀指令：</span><span class="si">{</span><span class="n">reason</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">175</span><span class="cl">    <span class="n">out</span><span class="p">[</span><span class="s2">&#34;updates&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln">176</span><span class="cl">        <span class="s2">&#34;mode&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">up</span><span class="p">,</span> <span class="s2">&#34;mode&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="s2">&#34;updates&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">177</span><span class="cl">        <span class="s2">&#34;timer&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">up</span><span class="p">,</span> <span class="s2">&#34;timer&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="s2">&#34;updates&#34;</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">178</span><span class="cl">        <span class="s2">&#34;config_files&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">str_list</span><span class="p">(</span><span class="n">up</span><span class="p">,</span> <span class="s2">&#34;config_files&#34;</span><span class="p">,</span> <span class="s2">&#34;updates&#34;</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> <span class="ow">or</span> <span class="p">[],</span>
</span></span><span class="line"><span class="ln">179</span><span class="cl">        <span class="s2">&#34;expect&#34;</span><span class="p">:</span> <span class="n">expect</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">180</span><span class="cl">        <span class="s2">&#34;reboot_check&#34;</span><span class="p">:</span> <span class="n">reboot</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">181</span><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="ln">182</span><span class="cl">
</span></span><span class="line"><span class="ln">183</span><span class="cl">    <span class="k">for</span> <span class="n">phase</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;suspend&#34;</span><span class="p">,</span> <span class="s2">&#34;resume&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">184</span><span class="cl">        <span class="n">steps</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">185</span><span class="cl">        <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">st</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">table_list</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">phase</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">186</span><span class="cl">            <span class="n">w</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">phase</span><span class="si">}</span><span class="s2">[</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s2">]&#34;</span>
</span></span><span class="line"><span class="ln">187</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln">188</span><span class="cl">                <span class="s2">&#34;step&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;step&#34;</span><span class="p">,</span> <span class="nb">int</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">189</span><span class="cl">                <span class="s2">&#34;resource&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;resource&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">190</span><span class="cl">                <span class="s2">&#34;billed_when_stopped&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;billed_when_stopped&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">BILLED_VALUES</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">191</span><span class="cl">                <span class="s2">&#34;billed_when_stopped_beyond_free&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;billed_when_stopped_beyond_free&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">BILLED_VALUES</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">192</span><span class="cl">                <span class="s2">&#34;billing_basis&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;billing_basis&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">193</span><span class="cl">                <span class="s2">&#34;where&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;where&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">enum</span><span class="o">=</span><span class="n">WHERE_VALUES</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">194</span><span class="cl">                <span class="s2">&#34;manual_command&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;manual_command&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">195</span><span class="cl">                <span class="s2">&#34;console_path&#34;</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="s2">&#34;console_path&#34;</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="n">w</span><span class="p">,</span> <span class="n">required</span><span class="o">=</span><span class="kc">False</span><span class="p">),</span>
</span></span><span class="line"><span class="ln">196</span><span class="cl">                <span class="s2">&#34;verify&#34;</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">197</span><span class="cl">            <span class="p">}</span>
</span></span><span class="line"><span class="ln">198</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">reject_old_command_name</span><span class="p">(</span><span class="n">st</span><span class="p">,</span> <span class="n">w</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">199</span><span class="cl">            <span class="k">if</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;manual_command&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;console_path&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">200</span><span class="cl">                <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;manual_command 或 console_path 至少要有一個&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">201</span><span class="cl">            <span class="n">vf</span> <span class="o">=</span> <span class="n">st</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;verify&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">202</span><span class="cl">            <span class="k">if</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">203</span><span class="cl">                <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">204</span><span class="cl">                    <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;where = </span><span class="se">\&#34;</span><span class="s2">host</span><span class="se">\&#34;</span><span class="s2"> 的 verify 要是只讀探針的 inline table&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">205</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">206</span><span class="cl">                    <span class="n">item</span><span class="p">[</span><span class="s2">&#34;verify&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">probe</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">w</span><span class="si">}</span><span class="s2">.verify&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">207</span><span class="cl">            <span class="k">elif</span> <span class="n">item</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">208</span><span class="cl">                <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">vf</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">209</span><span class="cl">                    <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s2">&#34;主機外的 verify 要是一句要看到的結果（字串）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">210</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">211</span><span class="cl">                    <span class="n">item</span><span class="p">[</span><span class="s2">&#34;verify&#34;</span><span class="p">]</span> <span class="o">=</span> <span class="n">v</span><span class="o">.</span><span class="n">subst</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">w</span><span class="si">}</span><span class="s2">.verify&#34;</span><span class="p">,</span> <span class="n">vf</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">212</span><span class="cl">            <span class="n">steps</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">213</span><span class="cl">        <span class="n">nums</span> <span class="o">=</span> <span class="p">[</span><span class="n">s</span><span class="p">[</span><span class="s2">&#34;step&#34;</span><span class="p">]</span> <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">steps</span> <span class="k">if</span> <span class="n">s</span><span class="p">[</span><span class="s2">&#34;step&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">214</span><span class="cl">        <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">nums</span><span class="p">)</span> <span class="o">!=</span> <span class="nb">len</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">nums</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">215</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">phase</span><span class="p">,</span> <span class="s2">&#34;step 有重複的編號&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">216</span><span class="cl">        <span class="n">out</span><span class="p">[</span><span class="n">phase</span><span class="p">]</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">steps</span><span class="p">,</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">s</span><span class="p">:</span> <span class="n">s</span><span class="p">[</span><span class="s2">&#34;step&#34;</span><span class="p">]</span> <span class="ow">or</span> <span class="mi">0</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">217</span><span class="cl">
</span></span><span class="line"><span class="ln">218</span><span class="cl">    <span class="n">_check_builtin_commands</span><span class="p">(</span><span class="n">v</span><span class="p">,</span> <span class="n">out</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">219</span><span class="cl">    <span class="k">if</span> <span class="n">v</span><span class="o">.</span><span class="n">errors</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">220</span><span class="cl">        <span class="k">raise</span> <span class="n">ManifestError</span><span class="p">(</span><span class="n">v</span><span class="o">.</span><span class="n">errors</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">221</span><span class="cl">    <span class="k">return</span> <span class="n">out</span>
</span></span><span class="line"><span class="ln">222</span><span class="cl">
</span></span><span class="line"><span class="ln">223</span><span class="cl">
</span></span><span class="line"><span class="ln">224</span><span class="cl"><span class="k">def</span> <span class="nf">_check_builtin_commands</span><span class="p">(</span><span class="n">v</span><span class="p">:</span> <span class="s2">&#34;_Validator&#34;</span><span class="p">,</span> <span class="n">out</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">225</span><span class="cl">    <span class="c1"># Checker 自己拼的指令也含清單的值（timer 名稱、journal unit），載入時一併比對白名單，不合就判成格式錯。</span>
</span></span><span class="line"><span class="ln">226</span><span class="cl">    <span class="n">cmds</span> <span class="o">=</span> <span class="p">[(</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">,</span> <span class="n">LISTEN_COMMAND</span><span class="p">)]</span>
</span></span><span class="line"><span class="ln">227</span><span class="cl">    <span class="n">timer</span> <span class="o">=</span> <span class="n">out</span><span class="p">[</span><span class="s2">&#34;updates&#34;</span><span class="p">][</span><span class="s2">&#34;timer&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">228</span><span class="cl">    <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">timer</span><span class="p">,</span> <span class="nb">str</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">229</span><span class="cl">        <span class="n">cmds</span> <span class="o">+=</span> <span class="p">[(</span><span class="s2">&#34;updates.timer&#34;</span><span class="p">,</span> <span class="n">timer_command</span><span class="p">(</span><span class="n">verb</span><span class="p">,</span> <span class="n">timer</span><span class="p">))</span> <span class="k">for</span> <span class="n">verb</span><span class="p">,</span> <span class="n">_</span> <span class="ow">in</span> <span class="n">UPDATE_TIMER_VERBS</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">230</span><span class="cl">    <span class="k">for</span> <span class="n">lg</span> <span class="ow">in</span> <span class="n">out</span><span class="p">[</span><span class="s2">&#34;log&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">231</span><span class="cl">        <span class="n">source</span> <span class="o">=</span> <span class="n">lg</span><span class="p">[</span><span class="s2">&#34;source&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">232</span><span class="cl">        <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">source</span><span class="p">,</span> <span class="nb">str</span><span class="p">)</span> <span class="ow">and</span> <span class="n">source</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;journal:&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">233</span><span class="cl">            <span class="n">cmds</span><span class="o">.</span><span class="n">append</span><span class="p">((</span><span class="sa">f</span><span class="s2">&#34;log[</span><span class="si">{</span><span class="n">lg</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">].source&#34;</span><span class="p">,</span> <span class="n">journal_command</span><span class="p">(</span><span class="n">source</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;:&#34;</span><span class="p">)[</span><span class="mi">2</span><span class="p">])))</span>
</span></span><span class="line"><span class="ln">234</span><span class="cl">    <span class="k">for</span> <span class="n">where</span><span class="p">,</span> <span class="n">cmd</span> <span class="ow">in</span> <span class="n">cmds</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">235</span><span class="cl">        <span class="n">reason</span> <span class="o">=</span> <span class="n">readonly_violation</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">236</span><span class="cl">        <span class="k">if</span> <span class="n">reason</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">237</span><span class="cl">            <span class="n">v</span><span class="o">.</span><span class="n">err</span><span class="p">(</span><span class="n">where</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;自動執行的指令不是只讀指令：</span><span class="si">{</span><span class="n">reason</span><span class="si">}</span><span class="s2">（</span><span class="si">{</span><span class="n">cmd</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span></span></span></code></pre></div><p><code>load_manifest</code> 把兩種讀不進來的情形（檔案打不開、內容不是合法的 TOML）都轉成 <code>ManifestError</code>，跟欄位錯走同一個 exit 2。錯誤訊息寫「不是合法的 TOML 1.0」，因為示範清單描述的那台 Oracle Linux 10 主機上，Python 3.12 的 <code>tomllib</code> 只認 TOML 1.0：在開發機上用較新的 Python 驗證過的清單，到了只有 3.12 的主機上，1.1 才有的語法會在這裡失敗；兩個版本的差異與解析器的版本在清單寫成 TOML 的那篇概念文章的〈TOML 1.0 與 1.1 的差異與解析器的版本〉。</p>
<p><code>validate</code> 先處理 <code>[vars]</code> 與 <code>--set</code>：<code>--set</code> 給的名稱要已經寫在 <code>[vars]</code> 裡，打錯字的名稱算格式錯（<code>test_unknown_set_name_is_format_error</code>），不會被當成一個新的變數收下、而清單裡想覆寫的那個值照舊不變。同一份清單套到多台主機時，每台不同的值在執行時用 <code>--set</code> 覆寫、還是在部署時就寫進每台的清單，以及依角色過濾項目的取捨，在 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>。之後依清單的段落逐段驗：</p>
<ul>
<li><strong>必填與選填的段落</strong>：<code>[host]</code>、<code>[updates]</code>、<code>[[entrypoint]]</code>、<code>[[log]]</code>、<code>[[accumulation]]</code>、<code>[[suspend]]</code>、<code>[[resume]]</code> 缺了都算格式錯；<code>[[filesystem]]</code>（<code>table_list</code> 的 <code>required=False</code>）、<code>[vars]</code>（預設是空表格）與 <code>[listen_audit]</code>（預設寫在 <code>[listen_audit]</code> 那一項）可以不寫。所以一份只想檢查入口的清單，其他段落也要寫出來；這是這支腳本的規格，換成自己的腳本時依需要放寬。</li>
<li><strong><code>[listen_audit]</code></strong>：整段沒寫時，預設檢查 TCP、沒有額外允許的連接埠，沒寫這一段的清單照樣檢查有沒有多開的 TCP 連接埠；寫了這一段就要寫出 <code>protocols</code>，寫成空陣列等於關掉這項檢查。</li>
<li><strong>錯誤的位置</strong>：入口、log、會累積的資料有名稱時，錯誤的位置寫成 <code>entrypoint[https]</code> 這種帶名稱的寫法，名稱缺了才退回索引，改清單的人照名稱找得到那一段。</li>
<li><strong>入口的 <code>enforced_by</code> 與 <code>[[entrypoint.verify]]</code></strong>：<code>enforced_by</code> 的每一層至少要有一筆 <code>[[entrypoint.verify]]</code>，每一筆 <code>verify</code> 的 <code>layer</code> 都要在 <code>enforced_by</code> 裡；比對用的是集合，同一層寫兩筆 <code>verify</code> 是允許的。宣告由某一層擋、卻沒寫那一層怎麼確認，或寫了確認方式、卻沒宣告由那一層擋，兩種都是格式錯（<code>test_enforced_layer_without_verify</code> 測宣告了卻沒有確認的那一種）。理由是清單宣告的防護層，每一層都要有一個確認的動作，欄位這樣設計的推導在清單寫成 TOML 的那篇概念文章的〈清單欄位的設計：檢查腳本要讀的與人要讀的〉。</li>
<li><strong>會累積的資料</strong>：<code>path</code> 可以寫一個字串或一個陣列，先統一成陣列；<code>limit_bytes</code> 與 <code>retention_days</code> 至少要有一個，沒有門檻的累積項目只能印出大小、判定不了 <code>WARN</code>；<code>oldest_command</code> 用來從資料庫查最舊一筆的時間，它跟保留天數一起才有意義，也要過白名單。</li>
<li><strong>暫停與恢復的步驟</strong>：<code>verify</code> 的型別依 <code>where</code> 而定，在主機上確認的是一個只讀探針，寫成寫在同一行的表格（inline table），在主機外確認的是一句人要看到的結果；步驟編號重複時判成格式錯，之後依編號排序，清單裡的寫出順序不影響執行順序。</li>
</ul>
<p>最後 <code>_check_builtin_commands</code> 把腳本自己會組的指令（接聽連接埠的查詢、timer 的兩個查詢、journal 的查詢；journal 是 <a href="/blog/linux/dotfile/knowledge-cards/journald/" data-link-title="journald（systemd-journald）" data-link-desc="用 journalctl 查不到重開機之前的紀錄、journal 佔掉多少磁碟不清楚、某個服務的 log 出現 Suppressed 訊息，或要判斷一筆紀錄該去 journal 還是 /var/log 底下的檔案找時">journald（systemd-journald）</a> 收下的紀錄）用清單的值組出來比對白名單，載入時就擋下 <code>--host=evil</code> 這種 timer 名稱。全部驗完才看 <code>v.errors</code>，有任何一條就丟出 <code>ManifestError</code>。</p>
<h2 id="執行指令的-runner-與時鐘">執行指令的 Runner 與時鐘</h2>
<p>腳本執行外部程式都經過 <code>Runner.run</code>：執行之前再比對一次白名單，以固定的環境變數與逾時執行，找不到指令與逾時都轉成一個結果、不丟例外。單元測試把它換成假的執行器，記錄每一次被呼叫的指令並回傳預先寫好的結果，所以測試不碰主機的狀態；取現在時間的 <code>now_utc</code> 也可以替換，讓跟時間有關的判定在測試裡有固定的答案。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># ── 執行環境（測試時替換）─────────────────────────────────</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="nd">@dataclass</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="k">class</span> <span class="nc">Result</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    <span class="n">exit</span><span class="p">:</span> <span class="nb">int</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="n">stdout</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="n">stderr</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">
</span></span><span class="line"><span class="ln">10</span><span class="cl">
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="k">class</span> <span class="nc">Runner</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="s2">&#34;&#34;&#34;執行只讀指令。測試以假的 Runner 替換，記錄被呼叫的指令。&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">command</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">Result</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">        <span class="c1"># 執行前再比對一次白名單：validate 之後到執行之間沒有別的路徑能塞進指令，這是第二道。</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">        <span class="n">reason</span> <span class="o">=</span> <span class="n">readonly_violation</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">        <span class="k">if</span> <span class="n">reason</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">            <span class="k">raise</span> <span class="ne">RuntimeError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;拒絕執行非只讀指令：</span><span class="si">{</span><span class="n">reason</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="n">argv</span> <span class="o">=</span> <span class="n">shlex</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">command</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="n">env</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">environ</span><span class="p">,</span> <span class="n">LC_ALL</span><span class="o">=</span><span class="s2">&#34;C&#34;</span><span class="p">,</span> <span class="n">LANG</span><span class="o">=</span><span class="s2">&#34;C&#34;</span><span class="p">,</span> <span class="n">SYSTEMD_PAGER</span><span class="o">=</span><span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="n">PAGER</span><span class="o">=</span><span class="s2">&#34;cat&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">            <span class="n">p</span> <span class="o">=</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">argv</span><span class="p">,</span> <span class="n">capture_output</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">text</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="n">COMMAND_TIMEOUT_S</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="n">env</span><span class="p">,</span> <span class="n">stdin</span><span class="o">=</span><span class="n">subprocess</span><span class="o">.</span><span class="n">DEVNULL</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="k">except</span> <span class="ne">FileNotFoundError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">            <span class="k">return</span> <span class="n">Result</span><span class="p">(</span><span class="mi">127</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;找不到指令 </span><span class="si">{</span><span class="n">argv</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="k">except</span> <span class="n">subprocess</span><span class="o">.</span><span class="n">TimeoutExpired</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">            <span class="k">return</span> <span class="n">Result</span><span class="p">(</span><span class="mi">124</span><span class="p">,</span> <span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;超過 </span><span class="si">{</span><span class="n">COMMAND_TIMEOUT_S</span><span class="si">}</span><span class="s2"> 秒沒有結束&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="k">return</span> <span class="n">Result</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">returncode</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">stdout</span><span class="p">,</span> <span class="n">p</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">
</span></span><span class="line"><span class="ln">29</span><span class="cl">
</span></span><span class="line"><span class="ln">30</span><span class="cl"><span class="k">def</span> <span class="nf">now_utc</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">    <span class="k">return</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">(</span><span class="n">dt</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span></span></span></code></pre></div><p>執行前那一次比對不通過時，<code>Runner</code> 丟出 <code>RuntimeError</code>，命令列把它當成格式錯以 exit 2 結束（單元測試 <code>test_runner_refuses_non_readonly</code> 與 <code>test_runtime_error_becomes_exit_2</code>，本篇沒有收錄）。正常情形下載入清單時已經擋下，這一道防的是之後有人加了一條不經過 <code>validate</code> 就執行指令的程式路徑。</p>
<p><code>subprocess.run</code> 收的是切好的參數串列，不帶 <code>shell=True</code>，指令不經過 shell。<code>shlex.split</code> 只切詞，不做 shell 會做的展開：<code>*</code> 這類 glob、<code>~</code>、<code>$VAR</code> 都原樣傳給工具。清單寫 <code>du -s /var/log/x/*</code> 時，<code>du</code> 收到的是字面的 <code>/var/log/x/*</code>（實跑：<code>du</code> 以 exit 1 結束，錯誤是 <code>No such file</code>）；要量多個檔案的大小，寫在 <code>[[accumulation]]</code> 的 <code>path</code>，那裡的 glob 由 <code>expand_path</code> 展開。環境變數從呼叫者的環境複製一份再蓋掉四個：<code>LC_ALL=C</code> 與 <code>LANG=C</code> 讓工具印英文，<code>REBOOT_REQUIRED_TEXT</code> 與 <code>enabled</code>、<code>active</code> 這類比對字串才對得上；<code>SYSTEMD_PAGER</code> 設成空字串、<code>PAGER=cat</code>，<code>systemctl</code> 與 <code>journalctl</code> 就不會開分頁程式等人按鍵；<code>stdin</code> 接 <code>/dev/null</code>，要讀輸入的工具立刻讀到結尾。<code>PATH</code> 沿用呼叫者的：裸工具名依 <code>PATH</code> 找檔案，<code>PATH</code> 前面的目錄放了一個同名的檔案時，執行的是那一個、不是系統目錄裡的那一支，白名單的執行檔路徑那一層只看指令字串，擋不住這一種。所以腳本要經由 sudo（它的 <code>secure_path</code> 會重設 <code>PATH</code>）或 systemd 的 unit 執行：systemd 啟動 unit 的程序時不繼承下指令那個 shell 的環境變數，<code>PATH</code> 用 systemd 自己的預設值（systemd.exec(5) 的環境變數一節；本篇未實跑），條件寫在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉白名單三層比對裡的「執行檔的路徑」那一層。</p>
<p>找不到指令時回 127、逾時回 124，跟 shell 與 GNU <code>timeout</code> 用的 exit code 相同，錯誤訊息放在 <code>stderr</code>。工具自己也可能以 127 或 124 結束，所以 <code>Checker</code> 的 <code>run_probe</code> 同時看 exit code 與 <code>stderr</code> 是否以 <code>找不到指令</code> 或 <code>超過</code> 開頭，分辨這是 <code>Runner</code> 造出來的結果、還是工具自己的 exit code。</p>
<h2 id="checker-的各項檢查">Checker 的各項檢查</h2>
<p><code>Checker</code> 依清單逐項執行檢查，每一項產生一行報告：狀態（<code>PASS</code>、<code>WARN</code>、<code>MANUAL</code>）、項目名稱、理由，有量到數字的項目另外帶著那個數字給結果檔用。維護時跑入口、log、會累積的資料、檔案系統與自動更新五類，暫停與恢復時只跑那個階段的步驟。各類檢查要量什麼、門檻怎麼定，在 <a href="/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算</a>；輸出與 exit code 的讀法在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</p>
<h3 id="報告的一行與探針的判定">報告的一行與探針的判定</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># ── 檢查 ───────────────────────────────────────────────────</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">
</span></span><span class="line"><span class="ln">3</span><span class="cl">
</span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="nd">@dataclass</span>
</span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="k">class</span> <span class="nc">Line</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">6</span><span class="cl">    <span class="n">status</span><span class="p">:</span> <span class="nb">str</span>  <span class="c1"># PASS / WARN / MANUAL</span>
</span></span><span class="line"><span class="ln">7</span><span class="cl">    <span class="n">item</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="ln">8</span><span class="cl">    <span class="n">reason</span><span class="p">:</span> <span class="nb">str</span>
</span></span><span class="line"><span class="ln">9</span><span class="cl">    <span class="n">value</span><span class="p">:</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>  <span class="c1"># 量到的數字（探針值、累積的 bytes、可用 bytes）；結果檔拿它算與前一次的差</span></span></span></code></pre></div><p><code>Line</code> 的 <code>item</code> 是結果檔比對前後兩次的鍵（例如 <code>entrypoint/https:443/tcp</code>、<code>accumulation/journald</code>），由清單的名稱與欄位組出來；清單裡改了一個名稱，下一次的差異就印成舊名稱 <code>GONE</code>、新名稱 <code>NEW</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">class</span> <span class="nc">Checker</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">manifest</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">runner</span><span class="p">:</span> <span class="n">Runner</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">clock</span><span class="p">:</span> <span class="n">Callable</span><span class="p">[[],</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="p">]</span> <span class="o">=</span> <span class="n">now_utc</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">m</span> <span class="o">=</span> <span class="n">manifest</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">runner</span> <span class="o">=</span> <span class="n">runner</span> <span class="ow">or</span> <span class="n">Runner</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">clock</span> <span class="o">=</span> <span class="n">clock</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">lines</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="n">Line</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="k">def</span> <span class="nf">emit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">status</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">item</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">reason</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">value</span><span class="p">:</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">lines</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">Line</span><span class="p">(</span><span class="n">status</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="c1"># 探針：回傳 (ok, 理由, 量到的值)</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="k">def</span> <span class="nf">run_probe</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">p</span><span class="p">:</span> <span class="n">Probe</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runner</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">command</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">        <span class="n">out</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">        <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="ow">in</span> <span class="p">(</span><span class="mi">124</span><span class="p">,</span> <span class="mi">127</span><span class="p">)</span> <span class="ow">and</span> <span class="n">r</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">startswith</span><span class="p">((</span><span class="s2">&#34;找不到指令&#34;</span><span class="p">,</span> <span class="s2">&#34;超過&#34;</span><span class="p">)):</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">stderr</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">        <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_exit</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">            <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">!=</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_exit</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">                <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;exit </span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">exit</span><span class="si">}</span><span class="s2">（預期 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">expect_exit</span><span class="si">}</span><span class="s2">）</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span> <span class="ow">or</span> <span class="n">out</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="k">elif</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">p</span><span class="o">.</span><span class="n">ok_exit</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;exit </span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">exit</span><span class="si">}</span><span class="s2">（可接受 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">ok_exit</span><span class="si">}</span><span class="s2">）</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span> <span class="ow">or</span> <span class="n">out</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">        <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_stdout</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">out</span> <span class="o">!=</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;輸出 </span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">out</span><span class="p">)</span> <span class="ow">or</span> <span class="s1">&#39;（空）&#39;</span><span class="si">}</span><span class="s2">（預期 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">expect_stdout</span><span class="si">!r}</span><span class="s2">）&#34;</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">        <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_contains</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_contains</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;輸出沒有 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">expect_contains</span><span class="si">!r}</span><span class="s2">：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">out</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">        <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_not_contains</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">p</span><span class="o">.</span><span class="n">expect_not_contains</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;輸出含 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">expect_not_contains</span><span class="si">!r}</span><span class="s2">：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">out</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">        <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">measure</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">            <span class="k">return</span> <span class="kc">True</span><span class="p">,</span> <span class="n">_brief</span><span class="p">(</span><span class="n">out</span><span class="p">)</span> <span class="ow">or</span> <span class="sa">f</span><span class="s2">&#34;輸出為空，exit </span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">exit</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">            <span class="n">value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">measure</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">measure</span><span class="p">,</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">        <span class="k">except</span> <span class="ne">ValueError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">e</span><span class="p">),</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">        <span class="n">unit</span> <span class="o">=</span> <span class="s2">&#34;天&#34;</span> <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">measure</span> <span class="o">==</span> <span class="s2">&#34;age_days&#34;</span> <span class="k">else</span> <span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">        <span class="n">shown</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">value</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}{</span><span class="n">unit</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">measure</span> <span class="o">==</span> <span class="s2">&#34;age_days&#34;</span> <span class="k">else</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">value</span><span class="si">:</span><span class="s2">g</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">        <span class="k">if</span> <span class="n">p</span><span class="o">.</span><span class="n">warn_above</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">value</span> <span class="o">&gt;</span> <span class="n">p</span><span class="o">.</span><span class="n">warn_above</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">            <span class="k">return</span> <span class="kc">False</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;量到 </span><span class="si">{</span><span class="n">shown</span><span class="si">}</span><span class="s2">，超過門檻 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">warn_above</span><span class="si">:</span><span class="s2">g</span><span class="si">}{</span><span class="n">unit</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">value</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">        <span class="k">return</span> <span class="kc">True</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;量到 </span><span class="si">{</span><span class="n">shown</span><span class="si">}</span><span class="s2">（門檻 </span><span class="si">{</span><span class="n">p</span><span class="o">.</span><span class="n">warn_above</span><span class="si">:</span><span class="s2">g</span><span class="si">}{</span><span class="n">unit</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">,</span> <span class="n">value</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">
</span></span><span class="line"><span class="ln">40</span><span class="cl">    <span class="k">def</span> <span class="nf">measure</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">kind</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">stdout</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">float</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">        <span class="k">if</span> <span class="n">kind</span> <span class="o">==</span> <span class="s2">&#34;lines&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">            <span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="nb">sum</span><span class="p">(</span><span class="mi">1</span> <span class="k">for</span> <span class="n">ln</span> <span class="ow">in</span> <span class="n">stdout</span><span class="o">.</span><span class="n">splitlines</span><span class="p">()</span> <span class="k">if</span> <span class="n">ln</span><span class="o">.</span><span class="n">strip</span><span class="p">()))</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">        <span class="n">text</span> <span class="o">=</span> <span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">        <span class="k">if</span> <span class="s2">&#34;=&#34;</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">45</span><span class="cl">            <span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">rsplit</span><span class="p">(</span><span class="s2">&#34;=&#34;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">1</span><span class="p">]</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">        <span class="k">if</span> <span class="n">kind</span> <span class="o">==</span> <span class="s2">&#34;value&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">47</span><span class="cl">            <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">48</span><span class="cl">                <span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">            <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">                <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;輸出不是數字：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">stdout</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span> <span class="kn">from</span> <span class="bp">None</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">        <span class="k">if</span> <span class="n">text</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;&#34;</span><span class="p">,</span> <span class="s2">&#34;n/a&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">52</span><span class="cl">            <span class="c1"># systemctl show 的時間屬性沒有值時印空值或 n/a（timer 從未觸發就是這樣）。</span>
</span></span><span class="line"><span class="ln">53</span><span class="cl">            <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;沒有時間值（</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">stdout</span><span class="p">)</span> <span class="ow">or</span> <span class="s1">&#39;空的輸出&#39;</span><span class="si">}</span><span class="s2">；timer 從未觸發過時就是這樣）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">        <span class="n">ts</span> <span class="o">=</span> <span class="n">parse_time</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">55</span><span class="cl">        <span class="k">if</span> <span class="n">ts</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">56</span><span class="cl">            <span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;輸出不是時間值：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">stdout</span><span class="p">)</span> <span class="ow">or</span> <span class="s1">&#39;（空）&#39;</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">57</span><span class="cl">        <span class="k">return</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span> <span class="o">-</span> <span class="n">ts</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">/</span> <span class="mi">86400</span></span></span></code></pre></div><p><code>run_probe</code> 依固定的順序判定一個探針，第一個不符的條件決定那一項的理由：<code>Runner</code> 造出來的找不到指令或逾時；exit code（寫了 <code>expect_exit</code> 就只比它，否則比 <code>ok_exit</code>）；<code>expect_stdout</code> 比對去掉前後空白的整段輸出；<code>expect_contains</code> 與 <code>expect_not_contains</code> 在原始輸出裡找子字串；最後才量數字。exit code 不符時，理由附上 <code>stderr</code>（沒有時附輸出）的開頭，<code>WARN</code> 那一行就帶著工具自己的錯誤訊息。</p>
<p><code>measure</code> 有三種量法。<code>lines</code> 數非空白的行，用在列出符合條件的紀錄、每筆一行的指令；<code>value</code> 與 <code>age_days</code> 先取最後一個 <code>=</code> 之後的字串，因為 <code>systemctl show -p NRestarts</code> 這類指令印的是 <code>NRestarts=0</code>。<code>age_days</code> 遇到空值或 <code>n/a</code> 時的理由是「沒有時間值」，跟解析失敗的「輸出不是時間值」分開：「沒有時間值」代表 timer 從來沒有觸發過，是主機的狀態；「輸出不是時間值」代表腳本讀不懂這個格式，是腳本要修的地方。單獨的 <code>0</code> 不在「沒有時間值」的條件裡，<code>measure(&quot;age_days&quot;, &quot;LastTriggerUSec=0&quot;)</code> 得到「輸出不是時間值」（實跑）。單元測試 <code>test_alert_age_days_never_triggered</code>（本篇沒有收錄）斷言 timer 從未觸發時的理由裡有「沒有時間值」。門檻的比較是「大於才 <code>WARN</code>」，量到的值等於門檻時是 <code>PASS</code>（<code>test_alert_value_at_threshold_passes</code>），清單裡的 <code>warn_above = 0</code> 因此讀成「出現任何一筆就 <code>WARN</code>」。只有量了數字的探針回傳那個值，其餘回傳 <code>None</code>。</p>
<h3 id="入口與接聽中的連接埠">入口與接聽中的連接埠</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl">    <span class="k">def</span> <span class="nf">check_entrypoints</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">        <span class="n">listening</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">        <span class="k">if</span> <span class="nb">any</span><span class="p">(</span><span class="n">e</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;entrypoint&#34;</span><span class="p">])</span> <span class="ow">or</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">][</span><span class="s2">&#34;protocols&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">            <span class="n">listening</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">listening_sockets</span><span class="p">()</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">        <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;entrypoint&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;entrypoint/</span><span class="si">{</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;port&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;protocol&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">            <span class="k">if</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">                <span class="k">if</span> <span class="n">listening</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s2">&#34;讀不到監聽中的 socket（ss 失敗）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">                    <span class="n">addrs</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">({</span><span class="n">a</span> <span class="k">for</span> <span class="p">(</span><span class="n">proto</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span> <span class="ow">in</span> <span class="n">listening</span> <span class="k">if</span> <span class="n">proto</span> <span class="o">==</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;protocol&#34;</span><span class="p">]</span> <span class="ow">and</span> <span class="n">port</span> <span class="o">==</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;port&#34;</span><span class="p">]})</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">                    <span class="k">if</span> <span class="n">addrs</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">                        <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;有程序在聽（</span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">addrs</span><span class="p">)</span><span class="si">}</span><span class="s2">），exposure=</span><span class="si">{</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;exposure&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">                    <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">                        <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;沒有程序在聽這個連接埠，exposure=</span><span class="si">{</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;exposure&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;MANUAL&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;主機外的入口，exposure=</span><span class="si">{</span><span class="n">e</span><span class="p">[</span><span class="s1">&#39;exposure&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">，逐層確認見下方&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">            <span class="k">for</span> <span class="n">vf</span> <span class="ow">in</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;verify&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">                <span class="n">vitem</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">item</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">vf</span><span class="p">[</span><span class="s1">&#39;layer&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">                <span class="k">if</span> <span class="n">vf</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">                    <span class="n">ok</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_probe</span><span class="p">(</span><span class="n">vf</span><span class="p">[</span><span class="s2">&#34;probe&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span> <span class="k">if</span> <span class="n">ok</span> <span class="k">else</span> <span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">vitem</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;MANUAL&#34;</span><span class="p">,</span> <span class="n">vitem</span><span class="p">,</span> <span class="n">_manual_text</span><span class="p">(</span><span class="n">vf</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">],</span> <span class="n">vf</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;console_path&#34;</span><span class="p">),</span> <span class="n">vf</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;manual_command&#34;</span><span class="p">),</span> <span class="n">vf</span><span class="p">[</span><span class="s2">&#34;expect&#34;</span><span class="p">]))</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="k">if</span> <span class="n">listening</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">audit_listening</span><span class="p">(</span><span class="n">listening</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="k">def</span> <span class="nf">listening_sockets</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">set</span><span class="p">[</span><span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="nb">int</span><span class="p">]]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">        <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runner</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">LISTEN_COMMAND</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">        <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">        <span class="n">socks</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">        <span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">splitlines</span><span class="p">():</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">            <span class="n">parts</span> <span class="o">=</span> <span class="n">line</span><span class="o">.</span><span class="n">split</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">            <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">5</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">                <span class="k">continue</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">            <span class="n">proto</span> <span class="o">=</span> <span class="n">parts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">            <span class="n">local</span> <span class="o">=</span> <span class="n">parts</span><span class="p">[</span><span class="mi">4</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">            <span class="n">addr</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">port</span> <span class="o">=</span> <span class="n">local</span><span class="o">.</span><span class="n">rpartition</span><span class="p">(</span><span class="s2">&#34;:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">            <span class="k">if</span> <span class="ow">not</span> <span class="n">port</span><span class="o">.</span><span class="n">isdigit</span><span class="p">()</span> <span class="ow">or</span> <span class="n">proto</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">PROTOCOL_VALUES</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">                <span class="k">continue</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">            <span class="n">socks</span><span class="o">.</span><span class="n">add</span><span class="p">((</span><span class="n">proto</span><span class="p">,</span> <span class="n">addr</span><span class="p">,</span> <span class="nb">int</span><span class="p">(</span><span class="n">port</span><span class="p">)))</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">        <span class="k">return</span> <span class="n">socks</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">
</span></span><span class="line"><span class="ln">45</span><span class="cl">    <span class="k">def</span> <span class="nf">audit_listening</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">listening</span><span class="p">:</span> <span class="nb">set</span><span class="p">[</span><span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">,</span> <span class="nb">int</span><span class="p">]])</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">        <span class="n">la</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;listen_audit&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">47</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">la</span><span class="p">[</span><span class="s2">&#34;protocols&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">48</span><span class="cl">            <span class="k">return</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">        <span class="n">known</span> <span class="o">=</span> <span class="p">{(</span><span class="n">e</span><span class="p">[</span><span class="s2">&#34;protocol&#34;</span><span class="p">],</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;port&#34;</span><span class="p">])</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;entrypoint&#34;</span><span class="p">]</span> <span class="k">if</span> <span class="n">e</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span><span class="p">}</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">        <span class="n">known</span> <span class="o">|=</span> <span class="p">{(</span><span class="n">a</span><span class="p">[</span><span class="s2">&#34;protocol&#34;</span><span class="p">],</span> <span class="n">a</span><span class="p">[</span><span class="s2">&#34;port&#34;</span><span class="p">])</span> <span class="k">for</span> <span class="n">a</span> <span class="ow">in</span> <span class="n">la</span><span class="p">[</span><span class="s2">&#34;allow&#34;</span><span class="p">]}</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="nb">any</span><span class="p">(</span><span class="n">p</span> <span class="ow">in</span> <span class="n">la</span><span class="p">[</span><span class="s2">&#34;protocols&#34;</span><span class="p">]</span> <span class="k">for</span> <span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">_</span><span class="p">)</span> <span class="ow">in</span> <span class="n">listening</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">52</span><span class="cl">            <span class="c1"># ss 成功卻一個都沒列出：多半是看不到主機的網路（PrivateNetwork=yes 的沙箱），跟「沒有多開」分不開。</span>
</span></span><span class="line"><span class="ln">53</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="s2">&#34;listen-audit&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;ss 沒有列出任何 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">la</span><span class="p">[</span><span class="s1">&#39;protocols&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2"> 的監聽 socket，判斷不了有沒有多開的連接埠（在 PrivateNetwork=yes 這類隔離網路的環境裡執行時就是這樣）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">            <span class="k">return</span>
</span></span><span class="line"><span class="ln">55</span><span class="cl">        <span class="n">extra</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">({(</span><span class="n">p</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span> <span class="k">for</span> <span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span> <span class="ow">in</span> <span class="n">listening</span> <span class="k">if</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">la</span><span class="p">[</span><span class="s2">&#34;protocols&#34;</span><span class="p">]</span> <span class="ow">and</span> <span class="ow">not</span> <span class="n">_is_loopback</span><span class="p">(</span><span class="n">a</span><span class="p">)</span> <span class="ow">and</span> <span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">port</span><span class="p">)</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">known</span><span class="p">})</span>
</span></span><span class="line"><span class="ln">56</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">extra</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">57</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="s2">&#34;listen-audit&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;loopback 以外只有清單列出的連接埠在聽（</span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">la</span><span class="p">[</span><span class="s1">&#39;protocols&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">58</span><span class="cl">        <span class="k">for</span> <span class="n">p</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">port</span> <span class="ow">in</span> <span class="n">extra</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">59</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;listen-audit/</span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">p</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">a</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">port</span><span class="si">}</span><span class="s2"> 在聽，但不在 [[entrypoint]] 或 [listen_audit].allow；用 ss -lnp</span><span class="si">{</span><span class="n">p</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="si">}</span><span class="s2"> 查是哪個程序&#34;</span><span class="p">)</span></span></span></code></pre></div><p><code>ss</code> 在一次維護檢查裡只執行一次，入口的判定與多開連接埠的稽核共用它的結果。<code>listening_sockets</code> 讀 <code>ss -H -l -n -t -u</code> 的每一行：第一欄是協定（<code>tcp</code>、<code>udp</code>），第五欄是本機的位址加連接埠；用 <code>rpartition(&quot;:&quot;)</code> 從右邊切，<code>[::]:22</code> 與 <code>*:8765</code> 這種含冒號或萬用字元的位址也切得出連接埠。<code>ss</code> 的 exit code 不是 0 時回傳 <code>None</code>，主機上的每個入口都印成「讀不到」的 <code>WARN</code>，不判成沒有程式在接聽。</p>
<p>主機上的入口只要有任何一個位址在接聽那個協定與連接埠，就印 <code>PASS</code>，理由裡列出接聽的位址與清單寫的 <code>exposure</code>。腳本不拿位址去比對 <code>exposure</code>：一個 <code>cdn-only</code> 的入口接聽在 <code>0.0.0.0</code>，對外的範圍由防火牆與雲端規則決定，那幾層的確認寫在 <code>[[entrypoint.verify]]</code>，在主機上的逐層跑探針，在主機外的印成 <code>MANUAL</code>。</p>
<p><code>audit_listening</code> 找清單沒有列出的連接埠：清單已經列出的連接埠是主機上的入口加 <code>[listen_audit].allow</code>，loopback 位址上的接聽不算多開，其餘每個接聽的位址加連接埠各印一行 <code>WARN</code>，項目名是 <code>listen-audit/&lt;連接埠&gt;/&lt;協定&gt;</code>，理由附上查出是哪個程式的 <code>ss</code> 指令。同一個連接埠同時接聽 <code>0.0.0.0</code> 與 <code>[::]</code> 時印兩行，兩行的項目名相同，結果檔只留後一行（〈host-check.py（3fc41c0）的已知限制與重現方式〉）。<code>ss</code> 成功而一個要稽核的協定的 socket 都沒有時印 <code>WARN</code>：在 <code>PrivateNetwork=yes</code> 的沙箱裡，<code>ss</code> 看不到主機的網路，只看結果分不出「沒有多開」與「什麼都看不到」，實跑的結果在清單寫成 TOML 的那篇概念文章的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉systemd 屬性表的 <code>PrivateNetwork=yes</code> 那一列。<code>test_extra_port_warns_and_missing_listener_warns</code> 給一份含 <code>0.0.0.0:3306</code> 與 <code>127.0.0.1:631</code> 的輸出，斷言 3306 印成 <code>WARN</code>、631 不出現，沒有程式接聽的 22 印成 <code>WARN</code>；<code>test_no_socket_listed_warns</code> 給空的輸出，斷言稽核那一行是 <code>WARN</code>。</p>
<h3 id="定期讀的-log">定期讀的 log</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl">    <span class="k">def</span> <span class="nf">check_logs</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">        <span class="k">for</span> <span class="n">lg</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;log&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;log/</span><span class="si">{</span><span class="n">lg</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">            <span class="n">kind</span><span class="p">,</span> <span class="n">_</span><span class="p">,</span> <span class="n">target</span> <span class="o">=</span> <span class="n">lg</span><span class="p">[</span><span class="s2">&#34;source&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;:&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">            <span class="n">hint</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;每 </span><span class="si">{</span><span class="n">lg</span><span class="p">[</span><span class="s1">&#39;interval_days&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> 天讀一次：</span><span class="si">{</span><span class="n">lg</span><span class="p">[</span><span class="s1">&#39;manual_command&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">            <span class="k">if</span> <span class="n">kind</span> <span class="o">==</span> <span class="s2">&#34;file&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">                <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">                    <span class="n">st</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">stat</span><span class="p">(</span><span class="n">target</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">                    <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">target</span><span class="p">,</span> <span class="s2">&#34;rb&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">                        <span class="k">pass</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">                <span class="k">except</span> <span class="ne">FileNotFoundError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">target</span><span class="si">}</span><span class="s2"> 不存在；</span><span class="si">{</span><span class="n">hint</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">                    <span class="k">continue</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">                <span class="k">except</span> <span class="ne">PermissionError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">target</span><span class="si">}</span><span class="s2"> 讀不到（權限，用 sudo 執行）；</span><span class="si">{</span><span class="n">hint</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">                    <span class="k">continue</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">target</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">st</span><span class="o">.</span><span class="n">st_size</span><span class="p">)</span><span class="si">}</span><span class="s2">；</span><span class="si">{</span><span class="n">hint</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">                <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runner</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">journal_command</span><span class="p">(</span><span class="n">target</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">                <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">==</span> <span class="mi">127</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span><span class="si">}</span><span class="s2">；</span><span class="si">{</span><span class="n">hint</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">                <span class="k">elif</span> <span class="ow">not</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;journal 裡沒有 </span><span class="si">{</span><span class="n">target</span><span class="si">}</span><span class="s2"> 的紀錄（unit 沒跑過，或沒有權限讀 journal）；</span><span class="si">{</span><span class="n">hint</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;最新一筆：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">(),</span> <span class="mi">90</span><span class="p">)</span><span class="si">}</span><span class="s2">；</span><span class="si">{</span><span class="n">hint</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">            <span class="k">for</span> <span class="n">al</span> <span class="ow">in</span> <span class="n">lg</span><span class="p">[</span><span class="s2">&#34;alerts&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">                <span class="n">ok</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_probe</span><span class="p">(</span><span class="n">al</span><span class="p">[</span><span class="s2">&#34;probe&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span> <span class="k">if</span> <span class="n">ok</span> <span class="k">else</span> <span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">item</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">al</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span></span></span></code></pre></div><p>讀 log 的內容是人的工作，腳本只確認 log 在那裡、讀得到，並在每一行的理由後面附上清單寫的讀取間隔與讀的指令，維護的人照那一行去讀。檔案型的 log 除了 <code>os.stat</code>，還實際開一次檔案：<code>stat</code> 只要求能進入上層目錄，檔案本身的讀取權限要開檔才知道。journal 型的 log 讀最新的一筆；輸出是空的有兩種成因（那個 unit 沒有跑過、執行者沒有讀 journal 的權限），腳本分不開，理由裡兩種都寫出來。這條路徑除了 127 不看 exit code：逾時（124）或 <code>journalctl</code> 以非零結束而沒有輸出時，印的也是同一句理由。log 底下的計數門檻（<code>[[log.alert]]</code>）各自是一個探針，照上面的 <code>run_probe</code> 判定。<code>test_missing_file_log_warns</code> 斷言檔案不存在的 log 印成 <code>WARN</code>。</p>
<h3 id="會累積的資料">會累積的資料</h3>
<p>這一段檢查清單的 <code>[[accumulation]]</code>，每一筆的欄位是 <code>name</code>、<code>path</code>（一個或多個路徑，可以是 glob 樣式）、<code>missing_ok</code>、<code>limit_bytes</code>、<code>retention_days</code>、<code>oldest_command</code>、<code>purge_command</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl">    <span class="k">def</span> <span class="nf">check_accumulations</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">        <span class="k">for</span> <span class="n">ac</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;accumulation&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;accumulation/</span><span class="si">{</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">            <span class="n">files</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">            <span class="n">denied</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">            <span class="k">for</span> <span class="n">pat</span> <span class="ow">in</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;paths&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">                <span class="n">found</span><span class="p">,</span> <span class="n">blocked</span> <span class="o">=</span> <span class="n">expand_path</span><span class="p">(</span><span class="n">pat</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">                <span class="n">files</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="n">found</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">                <span class="k">if</span> <span class="n">blocked</span> <span class="ow">and</span> <span class="n">blocked</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">denied</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">                    <span class="n">denied</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">blocked</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">            <span class="k">if</span> <span class="ow">not</span> <span class="n">files</span> <span class="ow">and</span> <span class="n">denied</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">                <span class="c1"># 讀不到不等於不存在：missing_ok 只管真的不存在的路徑。</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;讀不到 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">denied</span><span class="p">)</span><span class="si">}</span><span class="s2">（權限，用 sudo 執行），量不到 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;paths&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">                <span class="k">continue</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">            <span class="k">if</span> <span class="ow">not</span> <span class="n">files</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">                <span class="k">if</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;missing_ok&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;路徑不存在，算 0 bytes（</span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;paths&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;路徑不存在：</span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;paths&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">                <span class="k">continue</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">            <span class="n">total</span><span class="p">,</span> <span class="n">oldest</span><span class="p">,</span> <span class="n">unreadable</span> <span class="o">=</span> <span class="n">measure_paths</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">files</span><span class="p">)))</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">            <span class="n">problems</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">            <span class="n">facts</span> <span class="o">=</span> <span class="p">[</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">total</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">            <span class="k">if</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;limit_bytes&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">                <span class="n">facts</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">+=</span> <span class="sa">f</span><span class="s2">&#34;／上限 </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;limit_bytes&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">                <span class="k">if</span> <span class="n">total</span> <span class="o">&gt;</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;limit_bytes&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">                    <span class="n">problems</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;大小 </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">total</span><span class="p">)</span><span class="si">}</span><span class="s2"> 超過上限 </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;limit_bytes&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">            <span class="k">if</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;retention_days&#34;</span><span class="p">]</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">                <span class="k">if</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;oldest_command&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">                    <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runner</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">ac</span><span class="p">[</span><span class="s2">&#34;oldest_command&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">                    <span class="n">ts</span> <span class="o">=</span> <span class="n">parse_time</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">())</span> <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">==</span> <span class="mi">0</span> <span class="k">else</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">                    <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">                        <span class="n">problems</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;oldest_command exit </span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">exit</span><span class="si">}</span><span class="s2">：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">                    <span class="k">elif</span> <span class="n">ts</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">():</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">                        <span class="n">problems</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;oldest_command 的輸出不是時間值：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">                    <span class="n">oldest_ts</span> <span class="o">=</span> <span class="n">ts</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">                    <span class="n">label</span> <span class="o">=</span> <span class="s2">&#34;最舊一筆&#34;</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">                    <span class="n">oldest_ts</span> <span class="o">=</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">fromtimestamp</span><span class="p">(</span><span class="n">oldest</span><span class="p">,</span> <span class="n">dt</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span> <span class="k">if</span> <span class="n">oldest</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="k">else</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">                    <span class="n">label</span> <span class="o">=</span> <span class="s2">&#34;最舊的檔案&#34;</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">                <span class="k">if</span> <span class="n">oldest_ts</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">                    <span class="n">age</span> <span class="o">=</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">clock</span><span class="p">()</span> <span class="o">-</span> <span class="n">oldest_ts</span><span class="p">)</span><span class="o">.</span><span class="n">total_seconds</span><span class="p">()</span> <span class="o">/</span> <span class="mi">86400</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">                    <span class="n">facts</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">label</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">age</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> 天前／保留 </span><span class="si">{</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;retention_days&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> 天&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">                    <span class="k">if</span> <span class="n">age</span> <span class="o">&gt;</span> <span class="n">ac</span><span class="p">[</span><span class="s2">&#34;retention_days&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">45</span><span class="cl">                        <span class="n">problems</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">label</span><span class="si">}</span><span class="s2">是 </span><span class="si">{</span><span class="n">age</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> 天前，超過保留期 </span><span class="si">{</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;retention_days&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> 天&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">                <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">47</span><span class="cl">                    <span class="n">facts</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">label</span><span class="si">}</span><span class="s2">：沒有資料&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">48</span><span class="cl">            <span class="k">if</span> <span class="n">unreadable</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">                <span class="n">problems</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">unreadable</span><span class="si">}</span><span class="s2"> 個路徑讀不到（權限，用 sudo 執行）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">            <span class="k">if</span> <span class="n">denied</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">                <span class="n">problems</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;讀不到 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">denied</span><span class="p">)</span><span class="si">}</span><span class="s2">（權限，用 sudo 執行），那些路徑沒有算進大小&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">52</span><span class="cl">            <span class="k">if</span> <span class="n">problems</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">53</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s2">&#34;；&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">problems</span><span class="p">)</span> <span class="o">+</span> <span class="sa">f</span><span class="s2">&#34;。清理做法（不會自動執行）：</span><span class="si">{</span><span class="n">ac</span><span class="p">[</span><span class="s1">&#39;purge_command&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">total</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">55</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s2">&#34;，&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">facts</span><span class="p">),</span> <span class="n">total</span><span class="p">)</span></span></span></code></pre></div><p>一個累積項目的路徑可能是幾個 glob 樣式，每個樣式交給下面〈路徑的展開與大小的量測：expand_path 與 measure_paths〉的 <code>expand_path</code> 展開；展開時被權限擋下的目錄另外記在 <code>denied</code>。判定先分三種情形：一個檔案都沒找到而有目錄讀不到，印「讀不到」的 <code>WARN</code>；一個檔案都沒找到、也沒有目錄讀不到，才是路徑不存在，這時才看 <code>missing_ok</code>；找到檔案，量大小與時間。<code>missing_ok</code> 只管沒有被權限擋下的路徑（上層目錄列得出內容，或路徑根本不存在），讀不到的目錄即使設了 <code>missing_ok</code> 也印 <code>WARN</code>（<code>test_unreadable_glob_with_missing_ok_does_not_pass</code>）：資料可能就在那個讀不到的目錄裡，判成「不存在、算 0」會讓一個寫滿磁碟的目錄印成 <code>PASS</code>。</p>
<p>保留天數有兩種量法：寫了 <code>oldest_command</code> 時，執行它查資料最舊一筆的時間（例如 SQLite 裡某個時間欄位的最小值），沒有寫時取找到的檔案裡最舊的修改時間。<code>oldest_command</code> 成功而輸出是空的（資料表沒有資料）時，理由寫「沒有資料」，不算問題；輸出是 <code>n/a</code> 或 <code>0</code> 時，<code>parse_time</code> 回 <code>None</code> 而輸出不是空的，記成「oldest_command 的輸出不是時間值」的問題、那一項印 <code>WARN</code>（實跑）。超過上限、超過保留期、讀不到的路徑各記一條問題，任何一條都讓這一項印成 <code>WARN</code>，理由後面附上清單的 <code>purge_command</code>；<code>purge_command</code> 只印不執行，<code>test_accumulation_over_limit_warns</code> 斷言 <code>WARN</code> 那一行印出了清理指令，而要被清理的檔案仍然存在。這一項的 <code>value</code> 是總位元組，下一次執行時印成 <code>DELTA</code>。</p>
<h3 id="檔案系統的剩餘空間">檔案系統的剩餘空間</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl">    <span class="k">def</span> <span class="nf">check_filesystems</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">        <span class="k">for</span> <span class="n">fs</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;filesystem&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;filesystem/</span><span class="si">{</span><span class="n">fs</span><span class="p">[</span><span class="s1">&#39;name&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">            <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">                <span class="n">st</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">statvfs</span><span class="p">(</span><span class="n">fs</span><span class="p">[</span><span class="s2">&#34;mount&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">            <span class="k">except</span> <span class="ne">OSError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;讀不到 </span><span class="si">{</span><span class="n">fs</span><span class="p">[</span><span class="s1">&#39;mount&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">：</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">                <span class="k">continue</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">            <span class="n">free</span> <span class="o">=</span> <span class="n">st</span><span class="o">.</span><span class="n">f_bavail</span> <span class="o">*</span> <span class="n">st</span><span class="o">.</span><span class="n">f_frsize</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">            <span class="n">size</span> <span class="o">=</span> <span class="n">st</span><span class="o">.</span><span class="n">f_blocks</span> <span class="o">*</span> <span class="n">st</span><span class="o">.</span><span class="n">f_frsize</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">            <span class="n">text</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">fs</span><span class="p">[</span><span class="s1">&#39;mount&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> 剩 </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">free</span><span class="p">)</span><span class="si">}</span><span class="s2">／共 </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">size</span><span class="p">)</span><span class="si">}</span><span class="s2">（下限 </span><span class="si">{</span><span class="n">_human</span><span class="p">(</span><span class="n">fs</span><span class="p">[</span><span class="s1">&#39;min_free_bytes&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">）&#34;</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span> <span class="k">if</span> <span class="n">free</span> <span class="o">&gt;=</span> <span class="n">fs</span><span class="p">[</span><span class="s2">&#34;min_free_bytes&#34;</span><span class="p">]</span> <span class="k">else</span> <span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">text</span><span class="p">,</span> <span class="n">free</span><span class="p">)</span></span></span></code></pre></div><p><code>os.statvfs</code> 讀檔案系統的區塊計數，不執行 <code>df</code>。剩餘空間取 <code>f_bavail</code>（一般帳號可用的區塊數），不取 <code>f_bfree</code>（全部空閒的區塊數）：ext4 這類檔案系統預留給 root 的區塊算在 <code>f_bfree</code> 裡，而服務多半以一般帳號寫資料，<code>f_bavail</code> 用完時那些服務就寫不進去。區塊數乘上 <code>f_frsize</code>（區塊的大小）得到位元組。這一項的 <code>value</code> 是可用位元組（傳給 <code>emit</code> 的最後一個參數 <code>free</code>），兩次之間少了多少，差異段印得出來。可用空間幾乎每一次執行都不同，所以這一項幾乎每次都印 <code>DELTA</code>，要讀的是括號裡的差值。<code>test_filesystem_value_is_free_bytes</code> 只斷言結果檔裡這一項的值是數字、而且大於 0。</p>
<h3 id="自動更新的排程設定與重開機">自動更新的排程、設定與重開機</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl">    <span class="k">def</span> <span class="nf">check_updates</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">        <span class="n">up</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="s2">&#34;updates&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">        <span class="k">for</span> <span class="n">verb</span><span class="p">,</span> <span class="n">want</span> <span class="ow">in</span> <span class="n">UPDATE_TIMER_VERBS</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">            <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runner</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">timer_command</span><span class="p">(</span><span class="n">verb</span><span class="p">,</span> <span class="n">up</span><span class="p">[</span><span class="s2">&#34;timer&#34;</span><span class="p">]))</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">            <span class="n">got</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="ow">or</span> <span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span> <span class="ow">or</span> <span class="sa">f</span><span class="s2">&#34;exit </span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">exit</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span> <span class="k">if</span> <span class="n">got</span> <span class="o">==</span> <span class="n">want</span> <span class="k">else</span> <span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;updates/</span><span class="si">{</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;timer&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">verb</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">got</span><span class="si">}</span><span class="s2">（預期 </span><span class="si">{</span><span class="n">want</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">        <span class="n">merged</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">        <span class="n">read_files</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">        <span class="k">for</span> <span class="n">pat</span> <span class="ow">in</span> <span class="n">up</span><span class="p">[</span><span class="s2">&#34;config_files&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">            <span class="k">for</span> <span class="n">path</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">glob</span><span class="o">.</span><span class="n">glob</span><span class="p">(</span><span class="n">pat</span><span class="p">))</span> <span class="k">if</span> <span class="n">glob</span><span class="o">.</span><span class="n">has_magic</span><span class="p">(</span><span class="n">pat</span><span class="p">)</span> <span class="k">else</span> <span class="p">[</span><span class="n">pat</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">                <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">isfile</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">                    <span class="k">continue</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">                <span class="n">cp</span> <span class="o">=</span> <span class="n">configparser</span><span class="o">.</span><span class="n">ConfigParser</span><span class="p">(</span><span class="n">interpolation</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">                <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">                    <span class="n">cp</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">                <span class="k">except</span> <span class="p">(</span><span class="n">configparser</span><span class="o">.</span><span class="n">Error</span><span class="p">,</span> <span class="ne">OSError</span><span class="p">,</span> <span class="ne">UnicodeDecodeError</span><span class="p">)</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">                    <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="s2">&#34;updates/config&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2"> 讀不到或格式錯：</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">                    <span class="k">continue</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">                <span class="n">read_files</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">                <span class="k">if</span> <span class="n">cp</span><span class="o">.</span><span class="n">has_section</span><span class="p">(</span><span class="s2">&#34;commands&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">                    <span class="n">merged</span><span class="o">.</span><span class="n">update</span><span class="p">({</span><span class="n">k</span><span class="p">:</span> <span class="n">v</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">cp</span><span class="o">.</span><span class="n">items</span><span class="p">(</span><span class="s2">&#34;commands&#34;</span><span class="p">)})</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">read_files</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="s2">&#34;updates/config&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;找不到 dnf-automatic 的設定檔（</span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;config_files&#39;</span><span class="p">])</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">            <span class="n">wrong</span> <span class="o">=</span> <span class="p">{</span><span class="n">k</span><span class="p">:</span> <span class="n">merged</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">k</span><span class="p">)</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">up</span><span class="p">[</span><span class="s2">&#34;expect&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="n">merged</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">k</span><span class="p">)</span> <span class="o">!=</span> <span class="n">v</span><span class="p">}</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="s2">&#34;updates/config&#34;</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">            <span class="k">if</span> <span class="n">wrong</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s2">&#34;；&#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;[commands] </span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">g</span><span class="si">!r}</span><span class="s2">（預期 </span><span class="si">{</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;expect&#39;</span><span class="p">][</span><span class="n">k</span><span class="p">]</span><span class="si">!r}</span><span class="s2">）&#34;</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">g</span> <span class="ow">in</span> <span class="n">wrong</span><span class="o">.</span><span class="n">items</span><span class="p">())</span> <span class="o">+</span> <span class="sa">f</span><span class="s2">&#34;，讀了 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">read_files</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s2">&#34;, &#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2"> = </span><span class="si">{</span><span class="n">v</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">for</span> <span class="n">k</span><span class="p">,</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">up</span><span class="p">[</span><span class="s2">&#34;expect&#34;</span><span class="p">]</span><span class="o">.</span><span class="n">items</span><span class="p">())</span> <span class="o">+</span> <span class="sa">f</span><span class="s2">&#34;（mode </span><span class="si">{</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;mode&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">，讀了 </span><span class="si">{</span><span class="s1">&#39;, &#39;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">read_files</span><span class="p">)</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">        <span class="k">if</span> <span class="n">up</span><span class="p">[</span><span class="s2">&#34;reboot_check&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">            <span class="n">r</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">runner</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">up</span><span class="p">[</span><span class="s2">&#34;reboot_check&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="s2">&#34;updates/reboot&#34;</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">            <span class="k">if</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="s2">&#34;不需要重開機&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">            <span class="k">elif</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">==</span> <span class="mi">1</span> <span class="ow">and</span> <span class="n">REBOOT_REQUIRED_TEXT</span> <span class="ow">in</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span> <span class="o">+</span> <span class="n">r</span><span class="o">.</span><span class="n">stderr</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;更新之後需要重開機（</span><span class="si">{</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;reboot_check&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">）：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="o">.</span><span class="n">strip</span><span class="p">(),</span> <span class="mi">120</span><span class="p">)</span><span class="si">}</span><span class="s2">；deploy.sh 不自動重開，排時間 sudo systemctl reboot&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">            <span class="k">elif</span> <span class="n">r</span><span class="o">.</span><span class="n">exit</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;reboot_check&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> exit 1，但輸出沒有 </span><span class="si">{</span><span class="n">REBOOT_REQUIRED_TEXT</span><span class="si">!r}</span><span class="s2">，判斷不了要不要重開機：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span> <span class="ow">or</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span> <span class="ow">or</span> <span class="s1">&#39;（沒有輸出）&#39;</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">up</span><span class="p">[</span><span class="s1">&#39;reboot_check&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> exit </span><span class="si">{</span><span class="n">r</span><span class="o">.</span><span class="n">exit</span><span class="si">}</span><span class="s2">：</span><span class="si">{</span><span class="n">_brief</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">stderr</span> <span class="ow">or</span> <span class="n">r</span><span class="o">.</span><span class="n">stdout</span><span class="p">)</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span></span></span></code></pre></div><p>自動更新分三項檢查。清單的 <code>[updates].mode</code> 不在其中：它只出現在 <code>updates/config</code> 那一行 <code>PASS</code> 的理由裡（<code>mode {up['mode']}</code>），腳本不拿它比對任何設定或輸出。timer 的兩個查詢比對 <code>systemctl</code> 印的整個字串：<code>is-enabled</code> 要印 <code>enabled</code>、<code>is-active</code> 要印 <code>active</code>，輸出是空的時，理由改用 <code>stderr</code> 或 exit code，<code>WARN</code> 那一行看得到 <code>systemctl</code> 回了什麼。</p>
<p>設定檔可以寫成 glob 樣式（主設定檔，加上放在附加目錄裡、依檔名順序蓋過主設定的片段檔，也就是 drop-in；這是 dnf 自己的設定目錄，不改主設定檔、用片段蓋過其中的值這個做法，跟 <a href="/blog/linux/dotfile/knowledge-cards/systemd-drop-in/" data-link-title="systemd drop-in（不改原檔的設定疊加）" data-link-desc="想覆寫套件裝的 systemd unit 又不被升級蓋掉、一次對所有 service 套一條設定、或搞不清 systemctl edit 改了哪個檔時讀">systemd drop-in（不改原檔的設定疊加）</a> 相同），展開之後依檔名排序、依序讀，<code>[commands]</code> 段的值後讀到的蓋過先讀到的。<code>ConfigParser</code> 關掉 interpolation：預設的寫法會把值裡的 <code>%</code> 當成變數替換的語法，設定值含一個不成語法的 <code>%</code> 時，<code>cp.items</code> 取值就丟出例外，而取值的那一行 <code>merged.update({k: v.strip() for k, v in cp.items(&quot;commands&quot;)})</code> 不在 <code>try</code> 裡（<code>try</code> 只包住 <code>cp.read</code>），整支腳本會中止。清單的 <code>expect</code> 只列要確認的鍵，設定檔裡其他的鍵不比對；理由裡列出讀了哪些檔案，比對錯了的時候，維護的人知道值是從哪一份讀來的。</p>
<p><code>reboot_check</code> 的 exit 1 有兩種成因：dnf 判定要重開機，以及 dnf 自己出錯（在唯讀的檔案系統上寫不了 log）。腳本因此同時看輸出裡有沒有 <code>Reboot is required</code>，有才印「需要重開機」，沒有時印「判斷不了」並附上 dnf 的錯誤訊息；兩種成因的實跑結果與沙箱的設定在清單寫成 TOML 的那篇概念文章的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。<code>test_reboot_required_message_warns_reboot</code>、<code>test_reboot_exit1_without_message_is_not_reboot</code> 與 <code>test_reboot_exit0_passes</code> 各給一種輸出，斷言三種理由。理由裡的 <code>deploy.sh</code> 是私有專案部署那台主機的腳本，它裝好自動更新、不自動重開機。</p>
<h3 id="暫停與恢復的步驟以及依-phase-選檢查">暫停與恢復的步驟，以及依 phase 選檢查</h3>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl">    <span class="k">def</span> <span class="nf">check_phase</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">phase</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">        <span class="k">for</span> <span class="n">st</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">m</span><span class="p">[</span><span class="n">phase</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">            <span class="n">item</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">phase</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">st</span><span class="p">[</span><span class="s1">&#39;step&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">/</span><span class="si">{</span><span class="n">st</span><span class="p">[</span><span class="s1">&#39;resource&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">            <span class="n">billing</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;停機計費：額度內 </span><span class="si">{</span><span class="n">st</span><span class="p">[</span><span class="s1">&#39;billed_when_stopped&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">，超出額度 </span><span class="si">{</span><span class="n">st</span><span class="p">[</span><span class="s1">&#39;billed_when_stopped_beyond_free&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">（</span><span class="si">{</span><span class="n">st</span><span class="p">[</span><span class="s1">&#39;billing_basis&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">）&#34;</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">            <span class="n">action</span> <span class="o">=</span> <span class="n">st</span><span class="p">[</span><span class="s2">&#34;manual_command&#34;</span><span class="p">]</span> <span class="ow">or</span> <span class="n">st</span><span class="p">[</span><span class="s2">&#34;console_path&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">            <span class="k">if</span> <span class="n">st</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">]</span> <span class="o">==</span> <span class="s2">&#34;host&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;MANUAL&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;在主機上做（腳本不執行）：</span><span class="si">{</span><span class="n">action</span><span class="si">}</span><span class="s2">；</span><span class="si">{</span><span class="n">billing</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">                <span class="n">ok</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">run_probe</span><span class="p">(</span><span class="n">st</span><span class="p">[</span><span class="s2">&#34;verify&#34;</span><span class="p">])</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;PASS&#34;</span> <span class="k">if</span> <span class="n">ok</span> <span class="k">else</span> <span class="s2">&#34;WARN&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">item</span><span class="si">}</span><span class="s2">/verify&#34;</span><span class="p">,</span> <span class="n">reason</span><span class="p">,</span> <span class="n">value</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">            <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">                <span class="bp">self</span><span class="o">.</span><span class="n">emit</span><span class="p">(</span><span class="s2">&#34;MANUAL&#34;</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">_manual_text</span><span class="p">(</span><span class="n">st</span><span class="p">[</span><span class="s2">&#34;where&#34;</span><span class="p">],</span> <span class="n">st</span><span class="p">[</span><span class="s2">&#34;console_path&#34;</span><span class="p">],</span> <span class="n">st</span><span class="p">[</span><span class="s2">&#34;manual_command&#34;</span><span class="p">],</span> <span class="n">st</span><span class="p">[</span><span class="s2">&#34;verify&#34;</span><span class="p">])</span> <span class="o">+</span> <span class="sa">f</span><span class="s2">&#34;；</span><span class="si">{</span><span class="n">billing</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">phase</span><span class="p">:</span> <span class="nb">str</span> <span class="o">=</span> <span class="s2">&#34;maintain&#34;</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="n">Line</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">        <span class="k">if</span> <span class="n">phase</span> <span class="o">==</span> <span class="s2">&#34;maintain&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">check_entrypoints</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">check_logs</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">check_accumulations</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">check_filesystems</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">check_updates</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="k">else</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">            <span class="bp">self</span><span class="o">.</span><span class="n">check_phase</span><span class="p">(</span><span class="n">phase</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">        <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">lines</span></span></span></code></pre></div><p>暫停與恢復的每一步都印成 <code>MANUAL</code>，理由寫出要做的動作與那項資源停機時的計費；動作由人執行，腳本不執行 <code>manual_command</code>。在主機上的步驟另外執行 <code>verify</code> 的只讀探針，多印一行 <code>PASS</code> 或 <code>WARN</code>：維護的人做完那一步再跑一次腳本，<code>verify</code> 那一行確認那一步是否做成。<code>test_host_step_runs_only_verify</code> 斷言假的執行器只收到 <code>verify</code> 的指令，沒有收到 <code>manual_command</code>。各步驟要做什麼、停機時哪些資源還在計費，在 <a href="/blog/devops/operations/09-host-lifecycle/suspend-and-resume-service/" data-link-title="服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序" data-link-desc="一台主機上的服務要暫停一段時間再恢復：只停服務、停止主機、終止主機保留設定、全部刪除靠 repo 重建各自保留什麼與付出什麼，怎麼依保留的月費、重建的成本與漂移量選擇，暫停期間仍在計費的資源（含在作業系統裡 shutdown 不停止計費），仍被 DNS 與 CDN 指向的結果（Cloudflare 521、522，釋放 IP 之後的懸空 DNS），恢復前要重新確認的漂移，以及依元件依賴排出的恢復順序與驗證">服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序</a>。</p>
<p><code>run</code> 依 <code>--phase</code> 選要跑的檢查。維護時五類檢查依固定順序執行，順序只決定報告裡各行的先後；結果檔以項目名稱比對前後兩次，不依賴行的順序。</p>
<h2 id="路徑的展開與大小的量測expand_path-與-measure_paths">路徑的展開與大小的量測：expand_path 與 measure_paths</h2>
<p>這一段給會累積的資料用：把清單的路徑樣式展開成實際的檔案，分出「不存在」與「讀不到」，再用 <code>os.lstat</code> 量總大小與最舊的修改時間，不執行 <code>du</code>：白名單裡的 <code>du</code> 是給清單的探針用的，腳本自己量大小時不呼叫它。分出讀不到的路徑，是為了讓一般帳號執行時讀不到的目錄印成 <code>WARN</code>，而不是被當成路徑不存在，改用專用帳號實跑時讀不到哪些目錄，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># ── 小工具 ─────────────────────────────────────────────────</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="k">def</span> <span class="nf">expand_path</span><span class="p">(</span><span class="n">pat</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">],</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="s2">&#34;&#34;&#34;展開 [[accumulation]] 的一個路徑，回傳 (找到的路徑, 擋住讀取的目錄)。
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="s2">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="s2">    glob 與 os.path.lexists 遇到沒有權限的目錄都安靜地當成沒有（非 root 讀 0711 的 /var/log/nginx、
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="s2">    0750 的 /var/lib/monitor-collector、0700 的家目錄底下），所以找不到時再查一次：
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="s2">    glob 查萬用字元之前那一層目錄列不列得出來，一般路徑查 lstat 是不是被權限擋下。
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="s2">    萬用字元出現在中間一層、而更深一層的目錄讀不到時，這裡查不出來，仍然算成不存在。
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="s2">    &#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="k">if</span> <span class="n">glob</span><span class="o">.</span><span class="n">has_magic</span><span class="p">(</span><span class="n">pat</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">        <span class="n">found</span> <span class="o">=</span> <span class="n">glob</span><span class="o">.</span><span class="n">glob</span><span class="p">(</span><span class="n">pat</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">        <span class="k">if</span> <span class="n">found</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">            <span class="k">return</span> <span class="n">found</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">        <span class="n">parts</span> <span class="o">=</span> <span class="n">pat</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">sep</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">        <span class="n">i</span> <span class="o">=</span> <span class="nb">next</span><span class="p">(</span><span class="n">i</span> <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">part</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="n">parts</span><span class="p">)</span> <span class="k">if</span> <span class="n">glob</span><span class="o">.</span><span class="n">has_magic</span><span class="p">(</span><span class="n">part</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">        <span class="n">base</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">sep</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">parts</span><span class="p">[:</span><span class="n">i</span><span class="p">])</span> <span class="ow">or</span> <span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">sep</span> <span class="k">if</span> <span class="n">pat</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">sep</span><span class="p">)</span> <span class="k">else</span> <span class="n">os</span><span class="o">.</span><span class="n">curdir</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">            <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">base</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">        <span class="k">except</span> <span class="ne">PermissionError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">            <span class="k">return</span> <span class="p">[],</span> <span class="n">base</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">            <span class="k">pass</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="k">return</span> <span class="p">[],</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="n">os</span><span class="o">.</span><span class="n">lstat</span><span class="p">(</span><span class="n">pat</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="k">except</span> <span class="ne">PermissionError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">        <span class="k">return</span> <span class="p">[],</span> <span class="n">pat</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">    <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">        <span class="k">return</span> <span class="p">[],</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">    <span class="k">return</span> <span class="p">[</span><span class="n">pat</span><span class="p">],</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">
</span></span><span class="line"><span class="ln">34</span><span class="cl">
</span></span><span class="line"><span class="ln">35</span><span class="cl"><span class="k">def</span> <span class="nf">measure_paths</span><span class="p">(</span><span class="n">paths</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span><span class="p">,</span> <span class="nb">int</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">    <span class="s2">&#34;&#34;&#34;回傳 (總 bytes, 最舊的 mtime, 讀不到的路徑數)；不跟隨 symlink，同一個 inode 只算一次。&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">    <span class="n">total</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">    <span class="n">oldest</span><span class="p">:</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">    <span class="n">unreadable</span> <span class="o">=</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">    <span class="n">seen</span><span class="p">:</span> <span class="nb">set</span><span class="p">[</span><span class="nb">tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">
</span></span><span class="line"><span class="ln">42</span><span class="cl">    <span class="k">def</span> <span class="nf">visit</span><span class="p">(</span><span class="n">p</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">        <span class="k">nonlocal</span> <span class="n">total</span><span class="p">,</span> <span class="n">oldest</span><span class="p">,</span> <span class="n">unreadable</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">        <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">45</span><span class="cl">            <span class="n">st</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">lstat</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">        <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">47</span><span class="cl">            <span class="n">unreadable</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">48</span><span class="cl">            <span class="k">return</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">        <span class="n">key</span> <span class="o">=</span> <span class="p">(</span><span class="n">st</span><span class="o">.</span><span class="n">st_dev</span><span class="p">,</span> <span class="n">st</span><span class="o">.</span><span class="n">st_ino</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">        <span class="k">if</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">seen</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">            <span class="k">return</span>
</span></span><span class="line"><span class="ln">52</span><span class="cl">        <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">53</span><span class="cl">        <span class="k">if</span> <span class="n">stat</span><span class="o">.</span><span class="n">S_ISREG</span><span class="p">(</span><span class="n">st</span><span class="o">.</span><span class="n">st_mode</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">            <span class="n">total</span> <span class="o">+=</span> <span class="n">st</span><span class="o">.</span><span class="n">st_blocks</span> <span class="o">*</span> <span class="mi">512</span> <span class="k">if</span> <span class="n">st</span><span class="o">.</span><span class="n">st_blocks</span> <span class="k">else</span> <span class="n">st</span><span class="o">.</span><span class="n">st_size</span>
</span></span><span class="line"><span class="ln">55</span><span class="cl">            <span class="n">oldest</span> <span class="o">=</span> <span class="n">st</span><span class="o">.</span><span class="n">st_mtime</span> <span class="k">if</span> <span class="n">oldest</span> <span class="ow">is</span> <span class="kc">None</span> <span class="k">else</span> <span class="nb">min</span><span class="p">(</span><span class="n">oldest</span><span class="p">,</span> <span class="n">st</span><span class="o">.</span><span class="n">st_mtime</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">56</span><span class="cl">        <span class="k">elif</span> <span class="n">stat</span><span class="o">.</span><span class="n">S_ISDIR</span><span class="p">(</span><span class="n">st</span><span class="o">.</span><span class="n">st_mode</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">57</span><span class="cl">            <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">58</span><span class="cl">                <span class="n">entries</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">59</span><span class="cl">            <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">60</span><span class="cl">                <span class="n">unreadable</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">61</span><span class="cl">                <span class="k">return</span>
</span></span><span class="line"><span class="ln">62</span><span class="cl">            <span class="k">for</span> <span class="n">name</span> <span class="ow">in</span> <span class="n">entries</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">63</span><span class="cl">                <span class="n">visit</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">name</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">64</span><span class="cl">
</span></span><span class="line"><span class="ln">65</span><span class="cl">    <span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">paths</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">66</span><span class="cl">        <span class="n">visit</span><span class="p">(</span><span class="n">p</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">67</span><span class="cl">    <span class="k">return</span> <span class="n">total</span><span class="p">,</span> <span class="n">oldest</span><span class="p">,</span> <span class="n">unreadable</span></span></span></code></pre></div><p><code>glob.glob</code> 遇到列不出內容的目錄時回傳空串列，不丟例外，結果跟「沒有符合的檔案」一樣。<code>expand_path</code> 因此在 glob 找不到時，取樣式裡第一個含萬用字元的那一層之前的目錄，試著列一次內容：被權限擋下就回傳那個目錄，讓呼叫的程式印「讀不到」；列得出來（目錄在、只是沒有符合的檔案）或目錄不存在，才算不存在。不含萬用字元的路徑用 <code>os.lstat</code> 查，上層目錄不讓進入時 <code>lstat</code> 丟出 <code>PermissionError</code>。docstring 寫了這個做法查不出的情形：萬用字元在中間一層、更深一層讀不到時，仍然算成不存在。單元測試 <code>test_glob_in_unlistable_dir_is_unreadable</code>（本篇沒有收錄）用權限 0311（擁有者可以進入、不能列出內容）的目錄重現一般帳號看 0711 的 <code>/var/log/nginx</code>，斷言理由是「讀不到」，同一個目錄擋住兩個樣式時只列一次；<code>test_literal_path_under_unreadable_parent_is_unreadable</code> 用權限 0000 的上層目錄測不含萬用字元的路徑。這幾項測試要以一般帳號執行，以 root 執行時權限擋不住讀取，測試會失敗。</p>
<p><code>measure_paths</code> 遞迴走過每個路徑，用 <code>lstat</code> 而不用 <code>stat</code>：符號連結本身不算進大小，也不會被跟進去，一個指向 <code>/</code> 的連結不會讓腳本量整個檔案系統。每個檔案以（裝置編號、inode 編號）為鍵只算一次，inode 是檔案系統裡代表一個檔案的節點，硬連結共用同一個 inode，兩個樣式找到同一個檔案時也只算一次。大小取 <code>st_blocks * 512</code>，也就是檔案實際佔用的磁碟區塊（<code>st_blocks</code> 以 512 位元組為單位），跟 <code>du</code> 量的是同一種值：稀疏檔案的長度可能遠大於它佔用的空間，而上限要比的是磁碟用量。量的過程中讀不到的路徑只計數，不中斷，呼叫的程式把計數寫進 <code>WARN</code> 的理由。</p>
<p><code>st_blocks</code> 是 0 時的退路，跟「上限要比的是磁碟用量」互相衝突。程式在 <code>st_blocks</code> 是 0 時改用檔案的長度 <code>st_size</code>；這個退路是為哪一種情形，原始碼與加入它的 commit 都沒有寫。而整個檔案都是空洞的稀疏檔案，<code>st_blocks</code> 正好是 0，會被算成它的長度，寫在〈host-check.py（3fc41c0）的已知限制與重現方式〉的〈各項檢查：ss 失敗、重複的項目名、loopback、特殊檔案、稀疏檔案與時間值〉。</p>
<h2 id="時間字串的解析與組輸出用的小函式">時間字串的解析與組輸出用的小函式</h2>
<p><code>parse_time</code> 把 log 計數門檻的探針（<code>[[log.alert]]</code> 裡 <code>measure = &quot;age_days&quot;</code> 的那些）與 <code>[[accumulation]]</code> 的 <code>oldest_command</code> 印出的時間字串轉成帶時區的時間，接受 systemd 的格式、Unix 時間戳與 ISO 8601；之後是判斷 loopback 位址、把位元組換成 KiB 與 MiB、把輸出壓成一行、組 <code>MANUAL</code> 理由的小函式。腳本第一次在雲端主機上執行時讀不懂 systemd 印的時間格式，那一次的結果與修正在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉裡記錄雲端主機第一次執行結果的那幾段。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># systemd 的時間格式：systemctl show 印時間屬性（LastTriggerUSec 這類）時用它，不理</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"># --timestamp=unix。例：&#39;Mon 2026-10-26 07:07:43 GMT&#39;、&#39;Mon 2026-10-26 15:07:43.250000 CST&#39;。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="c1"># 時區是縮寫：UTC、GMT 直接認；其他縮寫要等於這台主機目前時區的縮寫（time.tzname）才當成本地</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="c1"># 時間，認不得的不猜（回傳 None）。</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="n">SYSTEMD_TIME_RE</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">compile</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;^(?:[A-Z][a-z]</span><span class="si">{2}</span><span class="s2"> )?(\d</span><span class="si">{4}</span><span class="s2">-\d</span><span class="si">{2}</span><span class="s2">-\d</span><span class="si">{2}</span><span class="s2"> \d</span><span class="si">{2}</span><span class="s2">:\d</span><span class="si">{2}</span><span class="s2">:\d</span><span class="si">{2}</span><span class="s2">(?:\.\d{1,6})?) ([A-Za-z]{2,6}|[+-]\d</span><span class="si">{2}</span><span class="s2">:?\d</span><span class="si">{2}</span><span class="s2">)$&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="k">def</span> <span class="nf">_parse_systemd_time</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="n">m</span> <span class="o">=</span> <span class="n">SYSTEMD_TIME_RE</span><span class="o">.</span><span class="k">match</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">m</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">        <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">    <span class="n">clock</span><span class="p">,</span> <span class="n">zone</span> <span class="o">=</span> <span class="n">m</span><span class="o">.</span><span class="n">groups</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="n">naive</span> <span class="o">=</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="n">clock</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="k">if</span> <span class="n">zone</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;UTC&#34;</span><span class="p">,</span> <span class="s2">&#34;GMT&#34;</span><span class="p">,</span> <span class="s2">&#34;UCT&#34;</span><span class="p">,</span> <span class="s2">&#34;Z&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">        <span class="k">return</span> <span class="n">naive</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">tzinfo</span><span class="o">=</span><span class="n">dt</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="k">if</span> <span class="n">zone</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="ow">in</span> <span class="s2">&#34;+-&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">        <span class="k">return</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">clock</span><span class="si">}{</span><span class="n">zone</span><span class="p">[:</span><span class="mi">3</span><span class="p">]</span><span class="si">}</span><span class="s2">:</span><span class="si">{</span><span class="n">zone</span><span class="p">[</span><span class="o">-</span><span class="mi">2</span><span class="p">:]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">    <span class="k">if</span> <span class="n">zone</span> <span class="ow">in</span> <span class="n">time</span><span class="o">.</span><span class="n">tzname</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="c1"># 本地時區：naive 的 astimezone() 依這個程序的時區規則換算（含夏令時間）。</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="k">return</span> <span class="n">naive</span><span class="o">.</span><span class="n">astimezone</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">
</span></span><span class="line"><span class="ln">23</span><span class="cl">
</span></span><span class="line"><span class="ln">24</span><span class="cl"><span class="k">def</span> <span class="nf">parse_time</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">    <span class="n">text</span> <span class="o">=</span> <span class="n">text</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">text</span> <span class="ow">or</span> <span class="n">text</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;n/a&#34;</span><span class="p">,</span> <span class="s2">&#34;0&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="k">if</span> <span class="n">re</span><span class="o">.</span><span class="n">fullmatch</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;@?\d+(\.\d+)?&#34;</span><span class="p">,</span> <span class="n">text</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">        <span class="k">return</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">fromtimestamp</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s2">&#34;@&#34;</span><span class="p">)),</span> <span class="n">dt</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">    <span class="n">systemd</span> <span class="o">=</span> <span class="n">_parse_systemd_time</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">    <span class="k">if</span> <span class="n">systemd</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">        <span class="k">return</span> <span class="n">systemd</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">    <span class="c1"># 小數秒超過 6 位（Go 的 RFC3339Nano）時截到微秒，fromisoformat 才吃得下。</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">    <span class="n">t</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;(\.\d</span><span class="si">{6}</span><span class="s2">)\d+&#34;</span><span class="p">,</span> <span class="sa">r</span><span class="s2">&#34;\1&#34;</span><span class="p">,</span> <span class="n">text</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="s2">&#34; &#34;</span><span class="p">,</span> <span class="s2">&#34;T&#34;</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">    <span class="n">t</span> <span class="o">=</span> <span class="n">re</span><span class="o">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s2">&#34;Z$&#34;</span><span class="p">,</span> <span class="s2">&#34;+00:00&#34;</span><span class="p">,</span> <span class="n">t</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">        <span class="n">ts</span> <span class="o">=</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="o">.</span><span class="n">fromisoformat</span><span class="p">(</span><span class="n">t</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">    <span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">        <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">    <span class="k">return</span> <span class="n">ts</span> <span class="k">if</span> <span class="n">ts</span><span class="o">.</span><span class="n">tzinfo</span> <span class="k">else</span> <span class="n">ts</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">tzinfo</span><span class="o">=</span><span class="n">dt</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span></span></span></code></pre></div><p>systemd 的時間格式以時區縮寫結尾，而縮寫不唯一對應一個時區：<code>CST</code> 在 <code>Asia/Taipei</code> 是台灣的標準時間，在美國中部是另一個時區。<code>_parse_systemd_time</code> 因此只認 UTC 這一類縮寫，以及等於這台主機目前時區縮寫的那一個（<code>time.tzname</code>），也認 <code>+0800</code>、<code>+08:00</code> 這種數字的時區偏移，其餘回傳 <code>None</code>，讓那一項印成「輸出不是時間值」，不猜一個時區而算出差了幾小時的天數。單元測試 <code>test_alert_age_days_systemd_local_zone</code>（本篇沒有收錄）把時區設成 <code>Asia/Taipei</code>，斷言 <code>CST</code> 換算正確、不認得的 <code>XYZT</code> 判成不是時間值；<code>test_alert_age_days_systemd_format</code> 測 <code>GMT</code>、<code>UTC</code> 與帶小數秒的寫法。</p>
<p><code>parse_time</code> 接受三種格式：可以有 <code>@</code> 前綴的 Unix 時間戳（epoch 秒數）、systemd 的格式、ISO 8601。systemd 的格式要另外處理，因為 <code>systemctl show</code> 印時間屬性（<code>LastTriggerUSec</code> 這類）時不理 <code>--timestamp=unix</code>，印的是 systemd 自己的格式；程式碼註解記了這一點，<code>test_alert_age_days_systemd_format</code> 用的就是那種輸出。</p>
<p>單獨的 <code>0</code> 與 <code>n/a</code>，<code>parse_time</code> 回 <code>None</code>，<code>@0</code> 照樣當成 1970-01-01（實跑）。回 <code>None</code> 之後怎麼判由呼叫端決定，兩個呼叫端的判法不同：<code>measure</code> 在呼叫 <code>parse_time</code> 之前，先把空的輸出與 <code>n/a</code> 判成「沒有時間值」，<code>0</code> 不在其中，結果是「輸出不是時間值」；<code>check_accumulations</code> 只把空的輸出當成沒有資料，<code>oldest_command</code> 印 <code>n/a</code> 或 <code>0</code> 時印 <code>WARN</code>「輸出不是時間值」（兩者都實跑）。<code>measure</code> 與 <code>check_accumulations</code> 各自的結果寫在〈host-check.py（3fc41c0）的已知限制與重現方式〉的〈各項檢查：ss 失敗、重複的項目名、loopback、特殊檔案、稀疏檔案與時間值〉。</p>
<p>ISO 8601 的字串截掉超過微秒的小數秒，程式碼註解寫的理由是 <code>datetime.fromisoformat</code> 吃不下 Go 的 RFC3339Nano 印出的奈秒，那是 Python 3.10 以前的行為：實跑 Python 3.10.18 的 <code>fromisoformat</code> 對九位小數丟出 <code>ValueError: Invalid isoformat string</code>，3.11.13、3.12.11、3.15.0 都接受九位小數、自己截到微秒。腳本要求 3.11 以上，所以在它支援的版本上，這一行截斷是冗餘的，結果跟不截斷相同。</p>
<p>沒有時區的 ISO 時間當成 UTC：字串本身沒有時區，腳本沒有別的資訊可以判斷。資料來源寫進去的是不帶時區的本地時間時，算出的天數會差一個時區偏移（台北是 8 小時，約 0.33 天），保留期的判定在門檻附近會早或晚 8 小時翻轉成 <code>WARN</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">def</span> <span class="nf">_is_loopback</span><span class="p">(</span><span class="n">addr</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">bool</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">    <span class="n">a</span> <span class="o">=</span> <span class="n">addr</span><span class="o">.</span><span class="n">strip</span><span class="p">(</span><span class="s2">&#34;[]&#34;</span><span class="p">)</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s2">&#34;%&#34;</span><span class="p">,</span> <span class="mi">1</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s2">&#34;127.&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">a</span> <span class="o">==</span> <span class="s2">&#34;::1&#34;</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="k">def</span> <span class="nf">_human</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">int</span> <span class="o">|</span> <span class="nb">float</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="k">for</span> <span class="n">unit</span> <span class="ow">in</span> <span class="p">(</span><span class="s2">&#34;B&#34;</span><span class="p">,</span> <span class="s2">&#34;KiB&#34;</span><span class="p">,</span> <span class="s2">&#34;MiB&#34;</span><span class="p">,</span> <span class="s2">&#34;GiB&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">        <span class="k">if</span> <span class="nb">abs</span><span class="p">(</span><span class="n">n</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">1024</span> <span class="ow">or</span> <span class="n">unit</span> <span class="o">==</span> <span class="s2">&#34;GiB&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">            <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">n</span><span class="si">:</span><span class="s2">.0f</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">unit</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="n">unit</span> <span class="o">==</span> <span class="s2">&#34;B&#34;</span> <span class="k">else</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">n</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">unit</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">        <span class="n">n</span> <span class="o">/=</span> <span class="mi">1024</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">n</span><span class="si">:</span><span class="s2">.1f</span><span class="si">}</span><span class="s2"> GiB&#34;</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">
</span></span><span class="line"><span class="ln">13</span><span class="cl">
</span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="k">def</span> <span class="nf">_brief</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">limit</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">160</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">    <span class="n">one</span> <span class="o">=</span> <span class="s2">&#34; &#34;</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">text</span><span class="o">.</span><span class="n">split</span><span class="p">())</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="k">return</span> <span class="n">one</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">one</span><span class="p">)</span> <span class="o">&lt;=</span> <span class="n">limit</span> <span class="k">else</span> <span class="n">one</span><span class="p">[:</span> <span class="n">limit</span> <span class="o">-</span> <span class="mi">1</span><span class="p">]</span> <span class="o">+</span> <span class="s2">&#34;…&#34;</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">
</span></span><span class="line"><span class="ln">18</span><span class="cl">
</span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="k">def</span> <span class="nf">_manual_text</span><span class="p">(</span><span class="n">where</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">console_path</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">,</span> <span class="n">command</span><span class="p">:</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">,</span> <span class="n">expect</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">    <span class="n">place</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;console&#34;</span><span class="p">:</span> <span class="s2">&#34;主控台&#34;</span><span class="p">,</span> <span class="s2">&#34;external&#34;</span><span class="p">:</span> <span class="s2">&#34;主機外&#34;</span><span class="p">}[</span><span class="n">where</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="n">how</span> <span class="o">=</span> <span class="n">console_path</span> <span class="k">if</span> <span class="n">console_path</span> <span class="k">else</span> <span class="sa">f</span><span class="s2">&#34;執行 </span><span class="si">{</span><span class="n">command</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;[</span><span class="si">{</span><span class="n">place</span><span class="si">}</span><span class="s2">] </span><span class="si">{</span><span class="n">how</span><span class="si">}</span><span class="s2"> → 預期：</span><span class="si">{</span><span class="n">expect</span><span class="si">}</span><span class="s2">&#34;</span></span></span></code></pre></div><p><code>_is_loopback</code> 先去掉 IPv6 位址外面的方括號與 <code>%</code> 之後的介面名稱（<code>ss</code> 印 <code>[::1]:631</code> 或 <code>127.0.0.53%lo:53</code> 這種寫法），再比對 <code>127.</code> 開頭或 <code>::1</code>。<code>_human</code> 的單位是 1024 進位的 KiB、MiB、GiB，跟 <code>du -h</code>、<code>df -h</code> 的換算方式相同，報告裡的大小拿去跟這兩支工具的輸出對照時不必換算。<code>_brief</code> 把多行的輸出壓成一行並截到 160 個字元，報告每一項維持一行，用 <code>grep</code> 篩報告時一項就是一行。</p>
<h2 id="結果檔的讀寫與跟前一次的差異">結果檔的讀寫與跟前一次的差異</h2>
<p>結果檔讓每一次的報告能跟同一個階段的前一次比對（階段就是 <code>--phase</code> 的 <code>maintain</code>、<code>suspend</code>、<code>resume</code>）：讀前一次的紀錄、算出差異、印在報告裡，最後把這一次的紀錄寫回去。寫回的做法是同一個目錄裡的暫存檔、<code>fsync</code>、<code>os.replace</code>，再 <code>fsync</code> 目錄，保證中斷時舊的紀錄仍然完整，說明在程式碼之後。結果檔的格式、印出的差異與預設路徑放在 <code>/var/lib</code> 的理由，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c1"># ── 結果檔（與前一次的差）──────────────────────────────────</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="c1"># 腳本自己的結果檔，不是清單的一部分：清單只描述要檢查什麼，量到的值存在這裡。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="c1"># 唯一會寫的檔案，而且只寫這一個（同目錄暫存檔 + os.replace，結束時目錄裡只有它）。</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="k">def</span> <span class="nf">load_state</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="c1"># 讀結果檔，回傳 (內容, 要印的 note)。不存在回空內容且不印 note；壞掉或格式不認得當成沒有。</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">        <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">path</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">            <span class="n">data</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">f</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="k">except</span> <span class="ne">FileNotFoundError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">        <span class="k">return</span> <span class="p">{},</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="k">except</span> <span class="p">(</span><span class="ne">OSError</span><span class="p">,</span> <span class="ne">ValueError</span><span class="p">)</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>  <span class="c1"># JSONDecodeError、UnicodeDecodeError 都是 ValueError</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">        <span class="k">return</span> <span class="p">{},</span> <span class="sa">f</span><span class="s2">&#34;結果檔 </span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2"> 讀不到或壞掉（</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">），當成沒有前一次，這次會覆寫&#34;</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">or</span> <span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;format&#34;</span><span class="p">)</span> <span class="o">!=</span> <span class="n">STATE_FORMAT</span> <span class="ow">or</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">data</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;phases&#34;</span><span class="p">),</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">        <span class="k">return</span> <span class="p">{},</span> <span class="sa">f</span><span class="s2">&#34;結果檔 </span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2"> 的格式不認得，當成沒有前一次，這次會覆寫&#34;</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="k">return</span> <span class="n">data</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">
</span></span><span class="line"><span class="ln">19</span><span class="cl">
</span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="k">def</span> <span class="nf">previous_phase</span><span class="p">(</span><span class="n">state</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">],</span> <span class="n">phase</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">    <span class="n">prev</span> <span class="o">=</span> <span class="n">state</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;phases&#34;</span><span class="p">,</span> <span class="p">{})</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">phase</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">    <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">prev</span><span class="p">,</span> <span class="nb">dict</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">prev</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;finished_at&#34;</span><span class="p">),</span> <span class="nb">str</span><span class="p">)</span> <span class="ow">and</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">prev</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;items&#34;</span><span class="p">),</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="k">return</span> <span class="n">prev</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">
</span></span><span class="line"><span class="ln">26</span><span class="cl">
</span></span><span class="line"><span class="ln">27</span><span class="cl"><span class="k">def</span> <span class="nf">build_record</span><span class="p">(</span><span class="n">lines</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="n">Line</span><span class="p">],</span> <span class="n">counts</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">int</span><span class="p">],</span> <span class="n">manifest_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">finished_at</span><span class="p">:</span> <span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="k">return</span> <span class="p">{</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">        <span class="s2">&#34;finished_at&#34;</span><span class="p">:</span> <span class="n">finished_at</span><span class="o">.</span><span class="n">astimezone</span><span class="p">(</span><span class="n">dt</span><span class="o">.</span><span class="n">timezone</span><span class="o">.</span><span class="n">utc</span><span class="p">)</span><span class="o">.</span><span class="n">isoformat</span><span class="p">(),</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">        <span class="s2">&#34;manifest&#34;</span><span class="p">:</span> <span class="n">manifest_path</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">        <span class="s2">&#34;counts&#34;</span><span class="p">:</span> <span class="n">counts</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">        <span class="s2">&#34;items&#34;</span><span class="p">:</span> <span class="p">{</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="p">:</span> <span class="p">{</span><span class="s2">&#34;status&#34;</span><span class="p">:</span> <span class="n">ln</span><span class="o">.</span><span class="n">status</span><span class="p">,</span> <span class="s2">&#34;value&#34;</span><span class="p">:</span> <span class="n">ln</span><span class="o">.</span><span class="n">value</span><span class="p">}</span> <span class="k">for</span> <span class="n">ln</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">},</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">    <span class="p">}</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">
</span></span><span class="line"><span class="ln">35</span><span class="cl">
</span></span><span class="line"><span class="ln">36</span><span class="cl"><span class="k">def</span> <span class="nf">_num</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">float</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">    <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">n</span><span class="si">:</span><span class="s2">.0f</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="nb">float</span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="o">.</span><span class="n">is_integer</span><span class="p">()</span> <span class="k">else</span> <span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">n</span><span class="si">:</span><span class="s2">.2f</span><span class="si">}</span><span class="s2">&#34;</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">
</span></span><span class="line"><span class="ln">39</span><span class="cl">
</span></span><span class="line"><span class="ln">40</span><span class="cl"><span class="k">def</span> <span class="nf">_prev_value</span><span class="p">(</span><span class="n">entry</span><span class="p">:</span> <span class="n">Any</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">tuple</span><span class="p">[</span><span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">,</span> <span class="nb">float</span> <span class="o">|</span> <span class="kc">None</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">entry</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">        <span class="k">return</span> <span class="kc">None</span><span class="p">,</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">    <span class="n">value</span> <span class="o">=</span> <span class="n">entry</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;value&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">    <span class="n">ok</span> <span class="o">=</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="nb">float</span><span class="p">))</span> <span class="ow">and</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">bool</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">45</span><span class="cl">    <span class="k">return</span> <span class="n">entry</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;status&#34;</span><span class="p">),</span> <span class="p">(</span><span class="n">value</span> <span class="k">if</span> <span class="n">ok</span> <span class="k">else</span> <span class="kc">None</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">
</span></span><span class="line"><span class="ln">47</span><span class="cl">
</span></span><span class="line"><span class="ln">48</span><span class="cl"><span class="k">def</span> <span class="nf">diff_lines</span><span class="p">(</span><span class="n">prev</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span><span class="p">,</span> <span class="n">lines</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="n">Line</span><span class="p">],</span> <span class="n">state_path</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">    <span class="c1"># 與前一次的差：第一行 since:，其後每個差一行（CHANGED / DELTA / NEW / GONE）。</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">    <span class="k">if</span> <span class="n">prev</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">        <span class="k">return</span> <span class="p">[</span><span class="sa">f</span><span class="s2">&#34;since: 沒有前一次的結果（</span><span class="si">{</span><span class="n">state_path</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">52</span><span class="cl">    <span class="n">out</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
</span></span><span class="line"><span class="ln">53</span><span class="cl">    <span class="n">seen</span><span class="p">:</span> <span class="nb">set</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="nb">set</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">    <span class="k">for</span> <span class="n">ln</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">55</span><span class="cl">        <span class="n">seen</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">56</span><span class="cl">        <span class="k">if</span> <span class="n">ln</span><span class="o">.</span><span class="n">item</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">prev</span><span class="p">[</span><span class="s2">&#34;items&#34;</span><span class="p">]:</span>
</span></span><span class="line"><span class="ln">57</span><span class="cl">            <span class="n">out</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;NEW     </span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="si">}</span><span class="s2">  </span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">status</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">58</span><span class="cl">            <span class="k">continue</span>
</span></span><span class="line"><span class="ln">59</span><span class="cl">        <span class="n">old_status</span><span class="p">,</span> <span class="n">old_value</span> <span class="o">=</span> <span class="n">_prev_value</span><span class="p">(</span><span class="n">prev</span><span class="p">[</span><span class="s2">&#34;items&#34;</span><span class="p">][</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="p">])</span>
</span></span><span class="line"><span class="ln">60</span><span class="cl">        <span class="k">if</span> <span class="n">old_status</span> <span class="o">!=</span> <span class="n">ln</span><span class="o">.</span><span class="n">status</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">61</span><span class="cl">            <span class="n">out</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;CHANGED </span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="si">}</span><span class="s2">  </span><span class="si">{</span><span class="n">old_status</span><span class="si">}</span><span class="s2">→</span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">status</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">62</span><span class="cl">        <span class="k">if</span> <span class="n">old_value</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">ln</span><span class="o">.</span><span class="n">value</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span> <span class="ow">and</span> <span class="n">old_value</span> <span class="o">!=</span> <span class="n">ln</span><span class="o">.</span><span class="n">value</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">63</span><span class="cl">            <span class="n">delta</span> <span class="o">=</span> <span class="n">ln</span><span class="o">.</span><span class="n">value</span> <span class="o">-</span> <span class="n">old_value</span>
</span></span><span class="line"><span class="ln">64</span><span class="cl">            <span class="n">out</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;DELTA   </span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="si">}</span><span class="s2">  </span><span class="si">{</span><span class="n">_num</span><span class="p">(</span><span class="n">old_value</span><span class="p">)</span><span class="si">}</span><span class="s2"> → </span><span class="si">{</span><span class="n">_num</span><span class="p">(</span><span class="n">ln</span><span class="o">.</span><span class="n">value</span><span class="p">)</span><span class="si">}</span><span class="s2">（</span><span class="si">{</span><span class="s1">&#39;+&#39;</span> <span class="k">if</span> <span class="n">delta</span> <span class="o">&gt;</span> <span class="mi">0</span> <span class="k">else</span> <span class="s1">&#39;-&#39;</span><span class="si">}{</span><span class="n">_num</span><span class="p">(</span><span class="nb">abs</span><span class="p">(</span><span class="n">delta</span><span class="p">))</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">65</span><span class="cl">    <span class="n">out</span><span class="o">.</span><span class="n">extend</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;GONE    </span><span class="si">{</span><span class="n">item</span><span class="si">}</span><span class="s2">&#34;</span> <span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">prev</span><span class="p">[</span><span class="s2">&#34;items&#34;</span><span class="p">])</span> <span class="o">-</span> <span class="n">seen</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">66</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">out</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">67</span><span class="cl">        <span class="k">return</span> <span class="p">[</span><span class="sa">f</span><span class="s2">&#34;since: </span><span class="si">{</span><span class="n">prev</span><span class="p">[</span><span class="s1">&#39;finished_at&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">，與前一次相同&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln">68</span><span class="cl">    <span class="k">return</span> <span class="p">[</span><span class="sa">f</span><span class="s2">&#34;since: </span><span class="si">{</span><span class="n">prev</span><span class="p">[</span><span class="s1">&#39;finished_at&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="o">*</span><span class="n">out</span><span class="p">]</span></span></span></code></pre></div><p>讀結果檔的三個函式都把讀到的內容當成不可信的輸入：檔案不存在是第一次執行，不印說明；JSON 壞掉、編碼不對、讀不到，回傳一條 <code>note</code>（由 <code>main</code> 印到 stderr）並當成沒有前一次；最外層不是 <code>format</code> 等於 <code>STATE_FORMAT</code>（1）、帶 <code>phases</code> 的物件時，同樣當成沒有前一次。<code>previous_phase</code> 與 <code>_prev_value</code> 再逐層確認型別，值是布林時不拿來算差。手改過或被別的程式寫壞的結果檔，最多讓差異段印成「沒有前一次的結果」，不會讓腳本中止（單元測試 <code>test_corrupt_state_does_not_abort</code>，本篇沒有收錄）。</p>
<p><code>diff_lines</code> 以項目名稱比對前後兩次，所以清單改了名稱的項目印成一筆 <code>GONE</code> 加一筆 <code>NEW</code>。同一個項目狀態與數字都變了時，<code>CHANGED</code> 與 <code>DELTA</code> 各印一行；<code>DELTA</code> 只在兩次都有數字時印，<code>value</code> 是 <code>null</code> 的項目只會出現在 <code>CHANGED</code>。<code>GONE</code> 依名稱排序，同樣的兩份紀錄每次印出的差異段順序相同。<code>test_new_and_gone_items</code> 斷言清單加進一個累積項目之後印 <code>NEW     accumulation/b  PASS</code>，拿掉之後印 <code>GONE    accumulation/b</code>；<code>test_second_run_prints_changed_and_delta</code> 斷言一個計數門檻 100 的探針從 42 變成 50 時印 <code>DELTA   log/edge/status-429  42 → 50（+8）</code>，變成 500 時同時印 <code>CHANGED … PASS→WARN</code> 與 <code>DELTA … 50 → 500（+450）</code>，再變成 20 時印 <code>DELTA … 500 → 20（-480）</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">def</span> <span class="nf">_fsync_directory</span><span class="p">(</span><span class="n">directory</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">    <span class="c1"># rename 要等目錄項寫進磁碟才耐斷電；Windows 不能開目錄，略過。</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">os</span><span class="p">,</span> <span class="s2">&#34;O_DIRECTORY&#34;</span><span class="p">):</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">        <span class="k">return</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="n">fd</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="n">directory</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">O_RDONLY</span> <span class="o">|</span> <span class="n">os</span><span class="o">.</span><span class="n">O_DIRECTORY</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">        <span class="n">os</span><span class="o">.</span><span class="n">fsync</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">        <span class="n">os</span><span class="o">.</span><span class="n">close</span><span class="p">(</span><span class="n">fd</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">
</span></span><span class="line"><span class="ln">11</span><span class="cl">
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="k">def</span> <span class="nf">write_state</span><span class="p">(</span><span class="n">path</span><span class="p">:</span> <span class="nb">str</span><span class="p">,</span> <span class="n">state</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="nb">str</span> <span class="o">|</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">    <span class="c1"># 原子地寫結果檔；失敗回傳 note（不改變 exit code）。</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">    <span class="n">directory</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="n">path</span><span class="p">))</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">    <span class="n">tmp_path</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">        <span class="n">os</span><span class="o">.</span><span class="n">makedirs</span><span class="p">(</span><span class="n">directory</span><span class="p">,</span> <span class="n">mode</span><span class="o">=</span><span class="mo">0o700</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">        <span class="n">fd</span><span class="p">,</span> <span class="n">tmp_path</span> <span class="o">=</span> <span class="n">tempfile</span><span class="o">.</span><span class="n">mkstemp</span><span class="p">(</span><span class="n">prefix</span><span class="o">=</span><span class="s2">&#34;.last-&#34;</span><span class="p">,</span> <span class="n">suffix</span><span class="o">=</span><span class="s2">&#34;.tmp&#34;</span><span class="p">,</span> <span class="nb">dir</span><span class="o">=</span><span class="n">directory</span><span class="p">)</span>  <span class="c1"># mkstemp 建出來就是 0600</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">        <span class="k">with</span> <span class="n">os</span><span class="o">.</span><span class="n">fdopen</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="s2">&#34;w&#34;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s2">&#34;utf-8&#34;</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">            <span class="n">json</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">f</span><span class="p">,</span> <span class="n">ensure_ascii</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">indent</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">sort_keys</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">            <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s2">&#34;</span><span class="se">\n</span><span class="s2">&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">            <span class="n">f</span><span class="o">.</span><span class="n">flush</span><span class="p">()</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">            <span class="n">os</span><span class="o">.</span><span class="n">fsync</span><span class="p">(</span><span class="n">f</span><span class="o">.</span><span class="n">fileno</span><span class="p">())</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">        <span class="n">os</span><span class="o">.</span><span class="n">replace</span><span class="p">(</span><span class="n">tmp_path</span><span class="p">,</span> <span class="n">path</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="n">tmp_path</span> <span class="o">=</span> <span class="kc">None</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">        <span class="n">_fsync_directory</span><span class="p">(</span><span class="n">directory</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">    <span class="k">except</span> <span class="ne">OSError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">        <span class="k">return</span> <span class="sa">f</span><span class="s2">&#34;結果檔 </span><span class="si">{</span><span class="n">path</span><span class="si">}</span><span class="s2"> 寫不進去（</span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">）；這次的結果沒有存下來，報告與 exit code 不受影響&#34;</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">    <span class="k">finally</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">        <span class="k">if</span> <span class="n">tmp_path</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">            <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">                <span class="n">os</span><span class="o">.</span><span class="n">unlink</span><span class="p">(</span><span class="n">tmp_path</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">            <span class="k">except</span> <span class="ne">OSError</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">                <span class="k">pass</span>  <span class="c1"># 暫存檔已不在（沒建成功或已被取代），沒有東西要清</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">    <span class="k">return</span> <span class="kc">None</span></span></span></code></pre></div><p><code>write_state</code> 的寫法依序是：在結果檔的同一個目錄用 <code>tempfile.mkstemp</code> 建暫存檔（權限 0600），寫完內容之後 <code>fsync</code>（要求作業系統把檔案內容寫到磁碟），再用 <code>os.replace</code> 換上正式的檔名，最後 <code>fsync</code> 那個目錄。暫存檔要在同一個目錄，因為 <code>os.replace</code> 在同一個檔案系統裡才是一次完成的換名；目錄的 <code>fsync</code> 讓換名這件事本身也寫進磁碟，斷電之後正式的檔名指著的是舊的或新的完整內容，不會是寫到一半的檔案。<code>os.replace</code> 擋的是中斷，擋不住兩次執行同時寫：程式裡沒有檔案鎖（<code>flock</code> 這類），兩次同時跑時，各自讀到的是開始時的結果檔，後換名的那一次蓋掉先換名的那一次寫進的紀錄（讀碼）。<code>test_write_state_fsyncs_file_and_directory</code> 斷言 <code>fsync</code> 至少呼叫兩次（檔案一次、目錄一次）。</p>
<p><code>os.makedirs</code> 的 <code>mode=0o700</code> 只套在這次新建的最後一層目錄上（中間各層用預設的權限），已經存在的目錄不改權限；<code>exist_ok=True</code> 只讓目錄已經存在時不丟例外。所以由 systemd 的 <code>StateDirectory=</code> 建出的目錄，權限照 <code>StateDirectoryMode=</code> 設定，腳本不跟 systemd 爭這個值。寫入失敗時，<code>finally</code> 刪掉留下的暫存檔，函式回傳一條 <code>note</code>，命令列印到 stderr，exit code 照檢查的結果決定（<code>test_unwritable_state_keeps_exit_code</code>）；成功時目錄裡只剩結果檔（<code>test_first_run_writes_only_last_json</code>）。</p>
<h2 id="命令列與-exit-codemain">命令列與 exit code：main</h2>
<p><code>main</code> 是命令列的入口：解析參數、載入清單、執行檢查、印報告與差異、寫結果檔，最後依報告決定 exit code。exit code 的讀法，以及接上排程時為什麼 exit code 不是 0 就要告警，在 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉；跨主機執行時每台的 exit code 在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈以 -b 與 systemd-run 執行檢查腳本：每台的輸出與 ansible 的 exit code〉。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">argv</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">]</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">runner</span><span class="p">:</span> <span class="n">Runner</span> <span class="o">|</span> <span class="kc">None</span> <span class="o">=</span> <span class="kc">None</span><span class="p">,</span> <span class="n">out</span><span class="o">=</span><span class="kc">None</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl">    <span class="n">out</span> <span class="o">=</span> <span class="n">out</span> <span class="ow">or</span> <span class="n">sys</span><span class="o">.</span><span class="n">stdout</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl">    <span class="n">ap</span> <span class="o">=</span> <span class="n">argparse</span><span class="o">.</span><span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="s2">&#34;web-demo 主機的只讀維護檢查&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl">    <span class="n">ap</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">&#34;--manifest&#34;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span><span class="vm">__file__</span><span class="p">)),</span> <span class="s2">&#34;host.toml&#34;</span><span class="p">))</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl">    <span class="n">ap</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">&#34;--phase&#34;</span><span class="p">,</span> <span class="n">choices</span><span class="o">=</span><span class="n">PHASES</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">&#34;maintain&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">    <span class="n">ap</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">&#34;--state-file&#34;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="n">DEFAULT_STATE_FILE</span><span class="p">,</span> <span class="n">metavar</span><span class="o">=</span><span class="s2">&#34;PATH&#34;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="sa">f</span><span class="s2">&#34;結果檔路徑（預設 </span><span class="si">{</span><span class="n">DEFAULT_STATE_FILE</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">    <span class="n">ap</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">&#34;--no-state&#34;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s2">&#34;store_true&#34;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">&#34;不讀也不寫結果檔&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">    <span class="n">ap</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s2">&#34;--set&#34;</span><span class="p">,</span> <span class="n">action</span><span class="o">=</span><span class="s2">&#34;append&#34;</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="p">[],</span> <span class="n">metavar</span><span class="o">=</span><span class="s2">&#34;NAME=VALUE&#34;</span><span class="p">,</span> <span class="n">help</span><span class="o">=</span><span class="s2">&#34;覆寫 [vars] 的值&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">    <span class="n">args</span> <span class="o">=</span> <span class="n">ap</span><span class="o">.</span><span class="n">parse_args</span><span class="p">(</span><span class="n">argv</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">    <span class="n">overrides</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">    <span class="k">for</span> <span class="n">s</span> <span class="ow">in</span> <span class="n">args</span><span class="o">.</span><span class="n">set</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">        <span class="n">k</span><span class="p">,</span> <span class="n">sep</span><span class="p">,</span> <span class="n">val</span> <span class="o">=</span> <span class="n">s</span><span class="o">.</span><span class="n">partition</span><span class="p">(</span><span class="s2">&#34;=&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">        <span class="k">if</span> <span class="ow">not</span> <span class="n">sep</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;manifest error: --set 要寫成 NAME=VALUE：</span><span class="si">{</span><span class="n">s</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl">            <span class="k">return</span> <span class="mi">2</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">        <span class="n">overrides</span><span class="p">[</span><span class="n">k</span><span class="p">]</span> <span class="o">=</span> <span class="n">val</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">        <span class="n">manifest</span> <span class="o">=</span> <span class="n">load_manifest</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">manifest</span><span class="p">,</span> <span class="n">overrides</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">    <span class="k">except</span> <span class="n">ManifestError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">20</span><span class="cl">        <span class="k">for</span> <span class="n">msg</span> <span class="ow">in</span> <span class="n">e</span><span class="o">.</span><span class="n">errors</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;manifest error: </span><span class="si">{</span><span class="n">msg</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">22</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;manifest error: </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">e</span><span class="o">.</span><span class="n">errors</span><span class="p">)</span><span class="si">}</span><span class="s2"> 項，沒有執行任何檢查&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">23</span><span class="cl">        <span class="k">return</span> <span class="mi">2</span>
</span></span><span class="line"><span class="ln">24</span><span class="cl">    <span class="k">if</span> <span class="nb">hasattr</span><span class="p">(</span><span class="n">os</span><span class="p">,</span> <span class="s2">&#34;geteuid&#34;</span><span class="p">)</span> <span class="ow">and</span> <span class="n">os</span><span class="o">.</span><span class="n">geteuid</span><span class="p">()</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">25</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="s2">&#34;note: 不是 root，journal、audit.log 與 collector 的目錄可能讀不到（會印成 WARN）&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">26</span><span class="cl">    <span class="k">try</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">27</span><span class="cl">        <span class="n">lines</span> <span class="o">=</span> <span class="n">Checker</span><span class="p">(</span><span class="n">manifest</span><span class="p">,</span> <span class="n">runner</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">phase</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">28</span><span class="cl">    <span class="k">except</span> <span class="ne">RuntimeError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">29</span><span class="cl">        <span class="c1"># validate 已比對過 Checker 會拼的指令，正常路徑不該走到；保險：當格式錯處理。</span>
</span></span><span class="line"><span class="ln">30</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;manifest error: </span><span class="si">{</span><span class="n">e</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">31</span><span class="cl">        <span class="k">return</span> <span class="mi">2</span>
</span></span><span class="line"><span class="ln">32</span><span class="cl">    <span class="n">counts</span> <span class="o">=</span> <span class="p">{</span><span class="s2">&#34;PASS&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="s2">&#34;WARN&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span> <span class="s2">&#34;MANUAL&#34;</span><span class="p">:</span> <span class="mi">0</span><span class="p">}</span>
</span></span><span class="line"><span class="ln">33</span><span class="cl">    <span class="k">for</span> <span class="n">ln</span> <span class="ow">in</span> <span class="n">lines</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">34</span><span class="cl">        <span class="n">counts</span><span class="p">[</span><span class="n">ln</span><span class="o">.</span><span class="n">status</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</span></span><span class="line"><span class="ln">35</span><span class="cl">        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;</span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">status</span><span class="si">:</span><span class="s2">&lt;6</span><span class="si">}</span><span class="s2"> </span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">item</span><span class="si">}</span><span class="s2">  </span><span class="si">{</span><span class="n">ln</span><span class="o">.</span><span class="n">reason</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">out</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">36</span><span class="cl">    <span class="n">state</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="n">Any</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="ln">37</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_state</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">38</span><span class="cl">        <span class="n">state</span><span class="p">,</span> <span class="n">note</span> <span class="o">=</span> <span class="n">load_state</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">state_file</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">39</span><span class="cl">        <span class="k">if</span> <span class="n">note</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">40</span><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;note: </span><span class="si">{</span><span class="n">note</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">41</span><span class="cl">        <span class="k">for</span> <span class="n">text</span> <span class="ow">in</span> <span class="n">diff_lines</span><span class="p">(</span><span class="n">previous_phase</span><span class="p">(</span><span class="n">state</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">phase</span><span class="p">),</span> <span class="n">lines</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">state_file</span><span class="p">):</span>
</span></span><span class="line"><span class="ln">42</span><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="n">text</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">out</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">43</span><span class="cl">    <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;summary: </span><span class="si">{</span><span class="n">counts</span><span class="p">[</span><span class="s1">&#39;PASS&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> PASS, </span><span class="si">{</span><span class="n">counts</span><span class="p">[</span><span class="s1">&#39;WARN&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> WARN, </span><span class="si">{</span><span class="n">counts</span><span class="p">[</span><span class="s1">&#39;MANUAL&#39;</span><span class="p">]</span><span class="si">}</span><span class="s2"> MANUAL（phase </span><span class="si">{</span><span class="n">args</span><span class="o">.</span><span class="n">phase</span><span class="si">}</span><span class="s2">，</span><span class="si">{</span><span class="n">args</span><span class="o">.</span><span class="n">manifest</span><span class="si">}</span><span class="s2">）&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">out</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">44</span><span class="cl">    <span class="k">if</span> <span class="ow">not</span> <span class="n">args</span><span class="o">.</span><span class="n">no_state</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">45</span><span class="cl">        <span class="n">phases</span> <span class="o">=</span> <span class="nb">dict</span><span class="p">(</span><span class="n">state</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s2">&#34;phases&#34;</span><span class="p">,</span> <span class="p">{}))</span>
</span></span><span class="line"><span class="ln">46</span><span class="cl">        <span class="n">phases</span><span class="p">[</span><span class="n">args</span><span class="o">.</span><span class="n">phase</span><span class="p">]</span> <span class="o">=</span> <span class="n">build_record</span><span class="p">(</span><span class="n">lines</span><span class="p">,</span> <span class="n">counts</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">manifest</span><span class="p">,</span> <span class="n">now_utc</span><span class="p">())</span>
</span></span><span class="line"><span class="ln">47</span><span class="cl">        <span class="n">note</span> <span class="o">=</span> <span class="n">write_state</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">state_file</span><span class="p">,</span> <span class="p">{</span><span class="s2">&#34;format&#34;</span><span class="p">:</span> <span class="n">STATE_FORMAT</span><span class="p">,</span> <span class="s2">&#34;phases&#34;</span><span class="p">:</span> <span class="n">phases</span><span class="p">})</span>
</span></span><span class="line"><span class="ln">48</span><span class="cl">        <span class="k">if</span> <span class="n">note</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">49</span><span class="cl">            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&#34;note: </span><span class="si">{</span><span class="n">note</span><span class="si">}</span><span class="s2">&#34;</span><span class="p">,</span> <span class="n">file</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="p">)</span>
</span></span><span class="line"><span class="ln">50</span><span class="cl">    <span class="k">return</span> <span class="mi">1</span> <span class="k">if</span> <span class="n">counts</span><span class="p">[</span><span class="s2">&#34;WARN&#34;</span><span class="p">]</span> <span class="k">else</span> <span class="mi">0</span>
</span></span><span class="line"><span class="ln">51</span><span class="cl">
</span></span><span class="line"><span class="ln">52</span><span class="cl">
</span></span><span class="line"><span class="ln">53</span><span class="cl"><span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">&#34;__main__&#34;</span><span class="p">:</span>
</span></span><span class="line"><span class="ln">54</span><span class="cl">    <span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="n">main</span><span class="p">())</span></span></span></code></pre></div><p><code>main</code> 收 <code>argv</code>、<code>runner</code> 與 <code>out</code> 三個參數，單元測試直接呼叫它，傳入假的執行器並把報告寫進記憶體裡的字串，不必另起一個程序。<code>--manifest</code> 的預設是腳本所在目錄的 <code>host.toml</code>，腳本與清單放在同一個目錄時不用帶這個參數。exit 2 的出口有四個：<code>argparse</code> 遇到不認得的參數或不在 <code>PHASES</code> 裡的 <code>--phase</code>（<code>argparse</code> 自己以 2 結束）、<code>--set</code> 沒有寫成 <code>名稱=值</code>、清單格式錯、執行時白名單拒絕了一條指令；前三個出口在任何檢查執行之前，執行時的拒絕發生時，在它之前的指令已經執行過。走到執行時的拒絕時，<code>RuntimeError</code> 從 <code>Checker(...).run</code> 裡丟出來，<code>run</code> 沒有回傳，<code>lines</code> 沒有被設定，所以已經執行過的那些項目，報告行一行都不會印出，stderr 印一行 <code>manifest error:</code> 加上拒絕的理由。清單格式錯時，每一條錯誤印一行 <code>manifest error:</code>，最後一行寫出錯誤的條數與「沒有執行任何檢查」。<code>-h</code>／<code>--help</code> 由 <code>argparse</code> 印出用法之後以 exit 0 結束，一項檢查都不執行，也沒有 <code>summary:</code> 那一行（實跑）；這支腳本裡，exit 0 而沒有 <code>summary:</code> 的只有這一種（讀碼），跟中途中止的 exit 1 分得開。單元測試 <code>test_cli_format_error_exit_2</code>（本篇沒有收錄）把腳本當成獨立的程序執行一份 <code>name = 1</code> 的清單，斷言 exit code 是 2、stderr 有 <code>manifest error</code>。</p>
<p>不是 root 時，<code>main</code> 在 stderr 印一行 <code>note:</code>，不改變 exit code：哪些項目會讀不到，由那幾項自己印成 <code>WARN</code>。報告的順序是每一項一行、差異段、<code>summary:</code> 一行；結果檔在檢查跑完之後才讀，結果檔壞掉不影響檢查本身，寫結果檔排在印出 <code>summary:</code> 之後，寫入失敗時報告已經完整印出。exit code 只看 <code>WARN</code> 的數量，<code>MANUAL</code> 不影響它。<code>if __name__ == &quot;__main__&quot;</code> 讓這個檔案被當成模組載入時（部署時的 <code>validate</code> 只呼叫 <code>load_manifest</code>）不執行檢查。</p>
<p>照這支腳本的做法替自己的主機寫檢查腳本時，程式裡有幾處字串是寫給示範清單那台主機的，要換成自己的環境：</p>
<ul>
<li><strong><code>web-demo</code></strong>：docstring 的第一行與 <code>argparse</code> 的 <code>description</code>，寫的是示範主機上的服務名。</li>
<li><strong><code>collector</code>、<code>audit.log</code></strong>：docstring 的用法說明與只讀保證，以及 <code>main</code> 不是 root 時印的那一行 <code>note:</code>，寫的是示範主機上讀不到的目錄與檔案。</li>
<li><strong>OCI Console、Cloudflare dashboard</strong>：docstring 第二段，寫的是示範主機外的確認位置。</li>
<li><strong><code>dnf-automatic</code></strong>：<code>check_updates</code> 找不到設定檔時印的理由；自動更新用的是別的工具時，<code>REBOOT_REQUIRED_TEXT</code> 與 <code>config_files</code> 的讀法也要一起換。</li>
<li><strong><code>deploy.sh 不自動重開，排時間 sudo systemctl reboot</code></strong>：<code>check_updates</code> 判定需要重開機時印的理由，<code>deploy.sh</code> 是私有專案部署那台主機的腳本。</li>
<li><strong><code>/var/lib/host-check/last.json</code> 與 <code>host.toml</code></strong>：<code>DEFAULT_STATE_FILE</code> 是結果檔的預設路徑，<code>--manifest</code> 的預設是腳本所在目錄的 <code>host.toml</code>；路徑換了時，systemd 沙箱的 <code>StateDirectory=</code> 也要跟著換。</li>
</ul>
<h2 id="host-checkpy3fc41c0的已知限制與重現方式">host-check.py（3fc41c0）的已知限制與重現方式</h2>
<p>這一版的腳本沒有處理的行為，依腳本的結構分成四組：清單的驗證、指令的執行與 exit code、各項檢查、結果檔。每一條寫出觸發條件與結果，括號裡的「實跑」與「讀碼」照開場的定義讀。</p>
<p>實跑的方式以同一個連接埠在兩個位址接聽那一條為例。在放著 <code>host-check.py</code> 的目錄執行下面這段，假的執行器不管收到哪一條指令都回同一份 <code>ss</code> 的輸出；第一行印出兩筆 <code>('WARN', 'listen-audit/111/tcp')</code>，第二行印出結果檔的 <code>items</code> 只剩一個 <code>listen-audit/111/tcp</code>：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln"> 1</span><span class="cl">python3 - <span class="s">&lt;&lt;&#39;EOF&#39;
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="s">import importlib.util
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="s">spec = importlib.util.spec_from_file_location(&#34;host_check&#34;, &#34;host-check.py&#34;)
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="s">hc = importlib.util.module_from_spec(spec)
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="s">spec.loader.exec_module(hc)
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="s">class FakeRunner(hc.Runner):
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="s">    def run(self, command):
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="s">        out = &#34;tcp LISTEN 0 128 0.0.0.0:111 0.0.0.0:*\ntcp LISTEN 0 128 [::]:111 [::]:*\n&#34;
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="s">        return hc.Result(0, out, &#34;&#34;)
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="s">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="s">manifest = {&#34;listen_audit&#34;: {&#34;protocols&#34;: [&#34;tcp&#34;], &#34;allow&#34;: []}, &#34;entrypoint&#34;: []}
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="s">checker = hc.Checker(manifest, FakeRunner())
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="s">checker.check_entrypoints()
</span></span></span><span class="line"><span class="ln">18</span><span class="cl"><span class="s">print([(ln.status, ln.item) for ln in checker.lines])
</span></span></span><span class="line"><span class="ln">19</span><span class="cl"><span class="s">print(list(hc.build_record(checker.lines, {}, &#34;m&#34;, hc.now_utc())[&#34;items&#34;]))
</span></span></span><span class="line"><span class="ln">20</span><span class="cl"><span class="s">EOF</span></span></span></code></pre></div><p>例子裡直接組出 <code>Checker</code> 收的 dict，因為 <code>listen_audit</code> 與空的 <code>entrypoint</code> 在 <code>validate</code> 正規化之後，形狀跟 TOML 的寫法相同。其餘段落正規化之後的形狀不同：<code>[[accumulation]]</code> 的 <code>path</code> 變成 <code>paths</code>（一律是串列），<code>[[log]]</code> 多一個 <code>alerts</code> 串列，每一個探針換成 <code>Probe</code> 物件，<code>[[entrypoint.verify]]</code> 在主機上的確認變成帶 <code>probe</code> 鍵的 dict。照 TOML 的欄位名組 dict 交給 <code>check_accumulations</code>，得到 <code>KeyError: 'paths'</code>（實跑）。所以其餘各條的重現，先寫一份 TOML 清單，用 <code>hc.load_manifest(&lt;清單的路徑&gt;)</code> 取得正規化之後的 dict，再交給 <code>hc.Checker</code>、呼叫對應的方法（<code>check_logs</code>、<code>check_updates</code>、<code>check_accumulations</code>、<code>check_entrypoints</code>）；<code>_Validator.probe</code>、<code>load_state</code>、<code>measure_paths</code>、<code>_is_loopback</code>、<code>parse_time</code> 這些函式可以直接呼叫；〈結果檔：壞掉時的其他階段與兩次同時執行〉的那幾條用 <code>hc.main([..., &quot;--state-file&quot;, &lt;路徑&gt;], runner=&lt;假的執行器&gt;)</code> 從頭跑。</p>
<h3 id="清單的驗證不認得的鍵同時寫-expect_exit-與-ok_exitexpect-的大寫鍵">清單的驗證：不認得的鍵、同時寫 expect_exit 與 ok_exit、expect 的大寫鍵</h3>
<ul>
<li><strong><code>[updates].expect</code> 的鍵有大寫字母</strong>（實跑）：<code>ConfigParser</code> 預設把設定檔裡的鍵轉成小寫，<code>merged</code> 的鍵因此全是小寫；清單的 <code>expect</code> 寫 <code>Upgrade_Type = &quot;security&quot;</code> 時，查 <code>merged.get(&quot;Upgrade_Type&quot;)</code> 得到 <code>None</code>，那一項永遠印 <code>WARN</code>。實跑時同一份設定檔，<code>expect</code> 的鍵寫 <code>Upgrade_Type</code> 時印 <code>WARN</code>（理由是 <code>Upgrade_Type = None</code>），寫 <code>upgrade_type</code> 時印 <code>PASS</code>。dnf-automatic 的設定檔裡 <code>[commands]</code> 的鍵寫成小寫（<code>upgrade_type</code>、<code>apply_updates</code>），清單照設定檔的寫法寫小寫就不會碰到。</li>
<li><strong>探針同時寫 <code>expect_exit</code> 與 <code>ok_exit</code></strong>（實跑）：<code>validate</code> 不拒絕這種寫法，<code>run_probe</code> 只比 <code>expect_exit</code>，<code>ok_exit</code> 不起作用。實跑時 <code>expect_exit = 0</code>、<code>ok_exit = [0, 1]</code> 的探針沒有任何格式錯，指令以 exit 1 結束時判成不符（理由 <code>exit 1（預期 0）</code>）。</li>
<li><strong><code>validate</code> 不認得的鍵與只在 <code>[[log.alert]]</code> 生效的 <code>measure</code></strong>（實跑）：<code>validate</code> 只讀它認得的鍵，其餘的鍵不報錯、當成沒寫。打錯字的鍵因此安靜地失效：實跑時一個寫了 <code>expect_exit = 0</code> 與 <code>expect_stdot = &quot;x&quot;</code> 的探針沒有任何格式錯，<code>expect_stdout</code> 讀出來是 <code>None</code>，輸出不會被比對；<code>[[accumulation]]</code> 的 <code>missing_okk</code> 照同一個機制被當成沒寫，<code>missing_ok</code> 取預設的 <code>false</code>（讀碼）。<code>measure</code> 與 <code>warn_above</code> 也屬於這一類：它們只在 log 的計數門檻（<code>[[log.alert]]</code>）讀，入口與暫停、恢復步驟的 <code>verify</code> 探針上寫了這兩個欄位、同一個探針又另外寫了別的預期值時，兩個欄位被略過（實跑：<code>measure</code> 讀出來是 <code>None</code>）；只寫 <code>measure</code> 而沒有其他預期值時，判成「探針至少要有一個預期值」的格式錯（實跑）。</li>
</ul>
<h3 id="指令的執行與-exit-code沒有接住的例外">指令的執行與 exit code：沒有接住的例外</h3>
<ul>
<li>
<p><strong><code>check_logs</code> 與 <code>Runner.run</code> 沒有接住的例外</strong>：兩處的 <code>try</code> 只接幾種指定的例外，沒有列出的例外一路往上，Python 印出 traceback、以 exit 1 結束，stdout 沒有 <code>summary:</code> 那一行。觸發的情形有三種：</p>
<ul>
<li><code>[[log]]</code> 的 <code>source</code> 寫成 <code>file:&lt;目錄&gt;</code>：<code>os.stat</code> 成功，接著 <code>open()</code> 丟出 <code>IsADirectoryError</code>（實跑，從命令列跑整支腳本：exit 1，stdout 沒有 <code>summary:</code>）。</li>
<li><code>source</code> 寫成 <code>file:/etc/hosts/x</code>，把一個檔案當成目錄：<code>os.stat</code> 丟出 <code>NotADirectoryError</code>（實跑，呼叫 <code>check_logs</code>）。<code>check_logs</code> 只接 <code>OSError</code> 底下的 <code>FileNotFoundError</code> 與 <code>PermissionError</code>，<code>IsADirectoryError</code> 與 <code>NotADirectoryError</code> 都不是這兩個子類別。</li>
<li><code>PATH</code> 上找到的同名檔案沒有執行權限、而 <code>PATH</code> 上沒有另一個可執行的同名檔案：<code>subprocess.run</code> 丟出 <code>PermissionError</code>，<code>Runner.run</code> 只接 <code>FileNotFoundError</code> 與逾時（實跑：<code>PATH</code> 只放一個權限 0644 的 <code>id</code>，<code>Runner().run(&quot;id&quot;)</code> 丟出 <code>PermissionError</code>）；經由命令列時的 exit 1 是讀碼。</li>
</ul>
<p>這個 exit 1 跟「至少一項 <code>WARN</code>」的 exit 1 相同，接上排程或告警的程式只看 exit code 時分不出兩者，要再看 stdout 有沒有 <code>summary:</code> 那一行（docstring 寫的「沒有印出 summary 代表腳本中途中止」）。替代的做法是在 <code>main</code> 外面包一層 <code>try</code>／<code>except</code>，把沒有接住的例外改回一個獨立的 exit code，這一版沒有實作。</p>
</li>
</ul>
<h3 id="各項檢查ss-失敗重複的項目名loopback特殊檔案稀疏檔案與時間值">各項檢查：ss 失敗、重複的項目名、loopback、特殊檔案、稀疏檔案與時間值</h3>
<ul>
<li><strong><code>updates/config</code> 的項目名重複</strong>（實跑）：<code>config_files</code> 裡有一個檔案讀不到或格式錯時，<code>check_updates</code> 先以 <code>updates/config</code> 印一行 <code>WARN</code>，讀完其餘的檔案之後比對 <code>expect</code>，又以同一個項目名印一行。<code>build_record</code> 以項目名當鍵，結果檔只留後一行。實跑時一個沒有段名的設定檔加一個正確的設定檔：報告印 <code>WARN</code> 與 <code>PASS</code> 兩行 <code>updates/config</code>，結果檔裡這一項是 <code>PASS</code>，讀不到設定檔的那一筆 <code>WARN</code> 不在結果檔裡，下一次的差異段也比不到它。</li>
<li><strong>會累積的資料裡的目錄與特殊檔案</strong>（實跑）：<code>measure_paths</code> 只把一般檔案算進大小與最舊的修改時間，目錄本身的修改時間不算；socket 與 FIFO 既不算進大小，也不算成讀不到的路徑。實跑時一個只放了一個 FIFO 的目錄，量到的是總大小 0、沒有最舊的時間、讀不到的路徑 0 個；這種目錄設了 <code>retention_days</code> 時，理由印「最舊的檔案：沒有資料」（讀碼）。</li>
<li><strong>全是空洞的稀疏檔案</strong>（實跑）：<code>measure_paths</code> 在 <code>st_blocks</code> 是 0 時改用 <code>st_size</code>，整個檔案都是空洞的稀疏檔案因此被算成它的長度，跟用 <code>st_blocks</code> 量磁碟用量的做法相反。實跑時在 macOS 上用 <code>truncate</code> 建一個 10 MiB 的稀疏檔案，<code>st_blocks</code> 是 0，<code>measure_paths</code> 量到 10485760 位元組；檔案只要有一個資料區塊，<code>st_blocks</code> 就不是 0，量到的是佔用的區塊（讀碼）。這個退路的理由原始碼沒有寫，<code>limit_bytes</code> 因此可能被一個沒有佔用磁碟的檔案觸發 <code>WARN</code>。</li>
<li><strong><code>ss</code> 以非零結束</strong>（實跑）：<code>listening_sockets</code> 回 <code>None</code>，主機上的每個入口印「讀不到監聽中的 socket」的 <code>WARN</code>，<code>listen-audit</code> 那一行整個不印。實跑時清單有一個主機上的入口，報告只有那個入口的 <code>WARN</code>；清單沒有 <code>where = &quot;host&quot;</code> 的入口時，報告裡一行跟 <code>ss</code> 有關的都沒有。前一次的結果有 <code>listen-audit</code> 那一項時，這一次的差異段印 <code>GONE listen-audit</code>（讀碼）。</li>
<li><strong>同一個連接埠在兩個位址接聽</strong>（實跑）：<code>audit_listening</code> 依協定、位址、連接埠各印一行，項目名只帶連接埠與協定；<code>0.0.0.0:111</code> 與 <code>[::]:111</code> 同時接聽時印兩行 <code>WARN</code>，項目名都是 <code>listen-audit/111/tcp</code>。<code>build_record</code> 以項目名當鍵，結果檔裡只留後一行，實跑時 <code>items</code> 裡只有一個 <code>listen-audit/111/tcp</code>；下一次的差異比的是留下的那一行。</li>
<li><strong>IPv4 對映的 loopback 位址</strong>（實跑）：<code>_is_loopback</code> 只認 <code>127.</code> 開頭與 <code>::1</code>，<code>[::ffff:127.0.0.1]</code> 回 <code>False</code>。一支程式在 IPv6 的 socket 上綁定 <code>::ffff:127.0.0.1</code> 時，<code>listen-audit</code> 把它當成 loopback 以外的接聽、印成 <code>WARN</code>（讀碼）。</li>
<li><strong>時間值寫成 <code>0</code> 或 <code>n/a</code></strong>（實跑）：<code>parse_time</code> 對單獨的 <code>0</code> 與 <code>n/a</code> 回 <code>None</code>，兩個呼叫端的判法不同。<code>measure</code> 先把空的輸出與 <code>n/a</code> 判成「沒有時間值」，<code>0</code> 不在其中：<code>measure(&quot;age_days&quot;, &quot;0&quot;)</code> 與 <code>measure(&quot;age_days&quot;, &quot;LastTriggerUSec=0&quot;)</code> 都得到「輸出不是時間值」，那一項的理由指向腳本讀不懂格式。<code>check_accumulations</code> 只把空的輸出當成沒有資料：<code>oldest_command</code> 印 <code>n/a</code> 或 <code>0</code> 時，記一條「oldest_command 的輸出不是時間值」、那一項印 <code>WARN</code>，只有空的輸出印 <code>PASS</code>（「最舊一筆：沒有資料」）。重現時呼叫 <code>hc.Checker({}, &lt;假的執行器&gt;).measure(&quot;age_days&quot;, &quot;0&quot;)</code>，或讓假的執行器對 <code>oldest_command</code> 回 <code>n/a</code>，再呼叫 <code>check_accumulations</code>。</li>
</ul>
<h3 id="結果檔壞掉時的其他階段與兩次同時執行">結果檔：壞掉時的其他階段與兩次同時執行</h3>
<ul>
<li><strong>結果檔壞掉時，其他階段的紀錄一起消失</strong>（實跑）：<code>load_state</code> 讀到不是合法 JSON 的結果檔時回傳空的內容，<code>main</code> 寫回時只帶這一次的階段。實跑時用 <code>hc.main([..., &quot;--state-file&quot;, &lt;路徑&gt;, &quot;--phase&quot;, &lt;階段&gt;], runner=&lt;假的執行器&gt;)</code> 依序跑 <code>suspend</code> 與 <code>maintain</code>，結果檔的 <code>phases</code> 是 <code>maintain</code> 與 <code>suspend</code>；把結果檔改成一段沒有收尾的 JSON 之後再跑一次 <code>maintain</code>，stderr 印「讀不到或壞掉」的 <code>note:</code>，寫回之後 <code>phases</code> 只剩 <code>maintain</code>。</li>
<li><strong>兩次執行同時寫結果檔</strong>（讀碼）：<code>write_state</code> 沒有檔案鎖，兩次同時跑時各自讀到開始時的結果檔，後換名的那一次蓋掉先換名的那一次；兩次跑的是不同的 <code>--phase</code> 時，先寫完的那個階段的紀錄消失。替代的做法是在腳本外面用 <code>flock</code> 這類工具讓兩次執行排隊，這一版沒有實作。</li>
</ul>
]]></content:encoded></item><item><title>主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</title><link>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/</link><pubDate>Mon, 26 Oct 2026 00:00:00 +0800</pubDate><guid>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/</guid><description>&lt;p>這篇把 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算&lt;/a> 與 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/suspend-and-resume-service/" data-link-title="服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序" data-link-desc="一台主機上的服務要暫停一段時間再恢復：只停服務、停止主機、終止主機保留設定、全部刪除靠 repo 重建各自保留什麼與付出什麼，怎麼依保留的月費、重建的成本與漂移量選擇，暫停期間仍在計費的資源（含在作業系統裡 shutdown 不停止計費），仍被 DNS 與 CDN 指向的結果（Cloudflare 521、522，釋放 IP 之後的懸空 DNS），恢復前要重新確認的漂移，以及依元件依賴排出的恢復順序與驗證">服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序&lt;/a> 要檢查的項目，寫成一份機器讀得懂、人也讀得懂的資料檔，並用一台實際的主機示範：為什麼要從文件改成資料檔；這份清單對檔案格式有哪些需求，以及 JSON、&lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/yaml/" data-link-title="YAML" data-link-desc="讀到 .yaml 或 .yml 設定檔、一個沒加引號的值被程式讀成布林值或別的數字、同一份檔案換一個解析器結果不同，或要在 JSON、YAML、TOML 之間選設定檔格式時">YAML&lt;/a>、TOML 逐項比對的結果（含 YAML 與 TOML 隱式型別的逐行實測）；清單用到的 TOML 寫法（表格、表格陣列、巢狀的表格陣列、寫在同一行的表格、兩種字串與 regex 的跳脫）；TOML 1.0 與 1.1 的差異，以及解析器版本對不上時的兩種結果；欄位依讀者分成檢查腳本要讀的與人要讀的兩類，各自怎麼設計；最後是讀這份清單的只讀檢查腳本，它的白名單要擋下哪些寫法才能保證不改動受檢查的設定與服務、哪些寫入白名單管不到而要靠系統層限制、輸出與 exit code 怎麼讀、結果檔怎麼跟上一次比對，以及它第一次在真主機上跑出的結果。TOML 的語法速查在 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/toml/" data-link-title="TOML" data-link-desc="讀到 .toml 設定檔、要判斷一個值會被解析成什麼型別、要在 JSON、YAML、TOML 之間選設定檔格式，或解析器版本對不上而報錯時">TOML&lt;/a> 知識卡。示範用的清單（&lt;code>host.toml&lt;/code>）放在作者的私有專案裡，完整內容沒有公開，本篇摘錄其中的段落，網域、IP 與帳號相關的值在摘錄裡用佔位值；檢查腳本（&lt;code>host-check.py&lt;/code>）的完整原始碼與逐段說明在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制&lt;/a>。同一份清單要套用到好幾台主機時，每台相同、依角色不同、每台不同的值怎麼分層，以及用 Ansible 的範本或 Terraform 的輸出產生每台清單的做法，在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍&lt;/a>。&lt;/p>
&lt;h2 id="從維護文件改成資料檔">從維護文件改成資料檔&lt;/h2>
&lt;p>維護的項目寫在文件裡時，執行的人每次讀一遍、挑出這次要做的、照著執行；文件跟實際的檢查之間沒有任何機制保證一致：文件說 SSH 只放行管理者 IP，而沒有任何一條指令在檢查這件事。把項目寫成資料檔，同一份內容有兩個讀者：維護的人照註解讀每一段，檢查腳本照欄位執行主機上量得到的項目，把主機外的項目（雲端主控台上的規則、從另一台機器才量得到的入口）印成待人工確認。主機上量得到的那些項目，清單寫了什麼，腳本就檢查什麼；主機外的項目，腳本只負責每次都把它們列出來，確認仍然靠人。新增一個檢查項目，等於同時更新了文件與檢查。&lt;/p>
&lt;p>示範清單涵蓋維護與暫停兩篇的主要項目：對外入口與每一層的防護、定期讀的 log 與它的計數門檻、會累積的資料與上限、檔案系統的剩餘空間、自動更新的設定，以及暫停與恢復上線的每一步；會到期的外部設定（憑證效期、CDN 位址段、管理者 IP）分散在入口與恢復上線的段落裡，syslog 的檔案與暫存檔沒有收進這份清單。項目的種類多、每一種的欄位不同，而且大部分是同一種項目列很多筆：示範清單裡有五個入口、六份 log、六類累積的資料、十六個暫停與恢復的步驟。&lt;/p>
&lt;h2 id="主機維護清單對檔案格式的需求jsonyamltoml-的比對">主機維護清單對檔案格式的需求：JSON、YAML、TOML 的比對&lt;/h2>
&lt;p>從清單的用途推出它對格式的需求：&lt;/p>
&lt;ul>
&lt;li>&lt;strong>可以寫註解&lt;/strong>：清單的讀者有一個是人，每一段要寫出它是什麼、欄位怎麼讀；註解要跟資料寫在同一份檔案裡，分開放就會分歧。&lt;/li>
&lt;li>&lt;strong>型別由寫法決定&lt;/strong>：門檻是數字、regex 與路徑是字串、開關是布林值；一個值被解析器讀成錯的型別時，檢查會照錯的值執行，而且不報錯。&lt;/li>
&lt;li>&lt;strong>同一種項目可以列很多筆&lt;/strong>：每一筆有自己的一組欄位。&lt;/li>
&lt;li>&lt;strong>主機上不必另裝套件就能解析&lt;/strong>：檢查腳本在主機上執行，主機上已經有的直譯器要讀得懂它。&lt;/li>
&lt;li>&lt;strong>巢狀不深&lt;/strong>：清單的結構是「一類項目 → 很多筆 → 每筆幾個欄位 → 少數幾筆子項目」。&lt;/li>
&lt;/ul>
&lt;p>拿這幾項比對三種常見的設定檔格式：&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>需求&lt;/th>
 &lt;th>JSON&lt;/th>
 &lt;th>YAML&lt;/th>
 &lt;th>TOML&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>註解&lt;/td>
 &lt;td>不支援&lt;/td>
 &lt;td>支援&lt;/td>
 &lt;td>支援&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>型別由寫法決定&lt;/td>
 &lt;td>是，規則嚴格&lt;/td>
 &lt;td>不加引號的值由解析器推測，YAML 1.1 與 1.2 的規則不同&lt;/td>
 &lt;td>是；不屬於任何型別的寫法解析失敗，看起來像字串、卻剛好合乎某個型別的寫法照那個型別讀&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>同一種項目列很多筆&lt;/td>
 &lt;td>陣列&lt;/td>
 &lt;td>序列&lt;/td>
 &lt;td>表格陣列 &lt;code>[[名稱]]&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>主機上的 Python 解析&lt;/td>
 &lt;td>標準函式庫 &lt;code>json&lt;/code>&lt;/td>
 &lt;td>要另裝 PyYAML 這類第三方套件&lt;/td>
 &lt;td>3.11 起標準函式庫 &lt;code>tomllib&lt;/code>；3.10 以下要另裝 &lt;code>tomli&lt;/code>&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>巢狀很深時&lt;/td>
 &lt;td>括號多但結構清楚&lt;/td>
 &lt;td>縮排表示結構，深的時候最簡潔&lt;/td>
 &lt;td>表頭要寫完整路徑，深的時候冗長&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>型別由寫法決定這一項，兩種格式的差別要實測才看得清楚。用 PyYAML 6.0.3（實作 YAML 1.1）的 &lt;code>safe_load&lt;/code> 解析下面這幾行：&lt;/p>





&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="ln">1&lt;/span>&lt;span class="cl">&lt;span class="nt">country&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">no&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">2&lt;/span>&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">010&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">3&lt;/span>&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">port_map&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">22&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">22&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">4&lt;/span>&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">1.10&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">5&lt;/span>&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">on&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>得到的是 &lt;code>{'country': False, 'mode': 8, 'port_map': 1342, 'version': 1.1, True: True}&lt;/code>：&lt;code>no&lt;/code> 被讀成布林值、&lt;code>010&lt;/code> 被讀成八進位的 8、&lt;code>22:22&lt;/code> 被讀成六十進位的 1342、&lt;code>1.10&lt;/code> 被讀成浮點數 1.1，連鍵名 &lt;code>on&lt;/code> 都變成了布林值 &lt;code>True&lt;/code>。每一個值都解析成功，沒有任何錯誤訊息；&lt;code>no&lt;/code> 被讀成 &lt;code>false&lt;/code> 這一種，英文資料常稱為 Norway problem（挪威的國家代碼 &lt;code>NO&lt;/code> 變成布林值）。放回這份清單：&lt;code>firewall-cmd --query-service=https&lt;/code> 印出的是字串 &lt;code>yes&lt;/code>，清單用 YAML 寫成 &lt;code>expect_stdout: yes&lt;/code> 時，PyYAML 把值讀成布林值 &lt;code>True&lt;/code>，跟輸出的字串比對永遠不相等，這一項每次都報警告；&lt;code>billed_when_stopped: no&lt;/code> 也一樣被讀成 &lt;code>False&lt;/code>。&lt;/p></description><content:encoded><![CDATA[<p>這篇把 <a href="/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算</a> 與 <a href="/blog/devops/operations/09-host-lifecycle/suspend-and-resume-service/" data-link-title="服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序" data-link-desc="一台主機上的服務要暫停一段時間再恢復：只停服務、停止主機、終止主機保留設定、全部刪除靠 repo 重建各自保留什麼與付出什麼，怎麼依保留的月費、重建的成本與漂移量選擇，暫停期間仍在計費的資源（含在作業系統裡 shutdown 不停止計費），仍被 DNS 與 CDN 指向的結果（Cloudflare 521、522，釋放 IP 之後的懸空 DNS），恢復前要重新確認的漂移，以及依元件依賴排出的恢復順序與驗證">服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序</a> 要檢查的項目，寫成一份機器讀得懂、人也讀得懂的資料檔，並用一台實際的主機示範：為什麼要從文件改成資料檔；這份清單對檔案格式有哪些需求，以及 JSON、<a href="/blog/devops/infra/knowledge-cards/yaml/" data-link-title="YAML" data-link-desc="讀到 .yaml 或 .yml 設定檔、一個沒加引號的值被程式讀成布林值或別的數字、同一份檔案換一個解析器結果不同，或要在 JSON、YAML、TOML 之間選設定檔格式時">YAML</a>、TOML 逐項比對的結果（含 YAML 與 TOML 隱式型別的逐行實測）；清單用到的 TOML 寫法（表格、表格陣列、巢狀的表格陣列、寫在同一行的表格、兩種字串與 regex 的跳脫）；TOML 1.0 與 1.1 的差異，以及解析器版本對不上時的兩種結果；欄位依讀者分成檢查腳本要讀的與人要讀的兩類，各自怎麼設計；最後是讀這份清單的只讀檢查腳本，它的白名單要擋下哪些寫法才能保證不改動受檢查的設定與服務、哪些寫入白名單管不到而要靠系統層限制、輸出與 exit code 怎麼讀、結果檔怎麼跟上一次比對，以及它第一次在真主機上跑出的結果。TOML 的語法速查在 <a href="/blog/devops/infra/knowledge-cards/toml/" data-link-title="TOML" data-link-desc="讀到 .toml 設定檔、要判斷一個值會被解析成什麼型別、要在 JSON、YAML、TOML 之間選設定檔格式，或解析器版本對不上而報錯時">TOML</a> 知識卡。示範用的清單（<code>host.toml</code>）放在作者的私有專案裡，完整內容沒有公開，本篇摘錄其中的段落，網域、IP 與帳號相關的值在摘錄裡用佔位值；檢查腳本（<code>host-check.py</code>）的完整原始碼與逐段說明在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</a>。同一份清單要套用到好幾台主機時，每台相同、依角色不同、每台不同的值怎麼分層，以及用 Ansible 的範本或 Terraform 的輸出產生每台清單的做法，在 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>。</p>
<h2 id="從維護文件改成資料檔">從維護文件改成資料檔</h2>
<p>維護的項目寫在文件裡時，執行的人每次讀一遍、挑出這次要做的、照著執行；文件跟實際的檢查之間沒有任何機制保證一致：文件說 SSH 只放行管理者 IP，而沒有任何一條指令在檢查這件事。把項目寫成資料檔，同一份內容有兩個讀者：維護的人照註解讀每一段，檢查腳本照欄位執行主機上量得到的項目，把主機外的項目（雲端主控台上的規則、從另一台機器才量得到的入口）印成待人工確認。主機上量得到的那些項目，清單寫了什麼，腳本就檢查什麼；主機外的項目，腳本只負責每次都把它們列出來，確認仍然靠人。新增一個檢查項目，等於同時更新了文件與檢查。</p>
<p>示範清單涵蓋維護與暫停兩篇的主要項目：對外入口與每一層的防護、定期讀的 log 與它的計數門檻、會累積的資料與上限、檔案系統的剩餘空間、自動更新的設定，以及暫停與恢復上線的每一步；會到期的外部設定（憑證效期、CDN 位址段、管理者 IP）分散在入口與恢復上線的段落裡，syslog 的檔案與暫存檔沒有收進這份清單。項目的種類多、每一種的欄位不同，而且大部分是同一種項目列很多筆：示範清單裡有五個入口、六份 log、六類累積的資料、十六個暫停與恢復的步驟。</p>
<h2 id="主機維護清單對檔案格式的需求jsonyamltoml-的比對">主機維護清單對檔案格式的需求：JSON、YAML、TOML 的比對</h2>
<p>從清單的用途推出它對格式的需求：</p>
<ul>
<li><strong>可以寫註解</strong>：清單的讀者有一個是人，每一段要寫出它是什麼、欄位怎麼讀；註解要跟資料寫在同一份檔案裡，分開放就會分歧。</li>
<li><strong>型別由寫法決定</strong>：門檻是數字、regex 與路徑是字串、開關是布林值；一個值被解析器讀成錯的型別時，檢查會照錯的值執行，而且不報錯。</li>
<li><strong>同一種項目可以列很多筆</strong>：每一筆有自己的一組欄位。</li>
<li><strong>主機上不必另裝套件就能解析</strong>：檢查腳本在主機上執行，主機上已經有的直譯器要讀得懂它。</li>
<li><strong>巢狀不深</strong>：清單的結構是「一類項目 → 很多筆 → 每筆幾個欄位 → 少數幾筆子項目」。</li>
</ul>
<p>拿這幾項比對三種常見的設定檔格式：</p>
<table>
  <thead>
      <tr>
          <th>需求</th>
          <th>JSON</th>
          <th>YAML</th>
          <th>TOML</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>註解</td>
          <td>不支援</td>
          <td>支援</td>
          <td>支援</td>
      </tr>
      <tr>
          <td>型別由寫法決定</td>
          <td>是，規則嚴格</td>
          <td>不加引號的值由解析器推測，YAML 1.1 與 1.2 的規則不同</td>
          <td>是；不屬於任何型別的寫法解析失敗，看起來像字串、卻剛好合乎某個型別的寫法照那個型別讀</td>
      </tr>
      <tr>
          <td>同一種項目列很多筆</td>
          <td>陣列</td>
          <td>序列</td>
          <td>表格陣列 <code>[[名稱]]</code></td>
      </tr>
      <tr>
          <td>主機上的 Python 解析</td>
          <td>標準函式庫 <code>json</code></td>
          <td>要另裝 PyYAML 這類第三方套件</td>
          <td>3.11 起標準函式庫 <code>tomllib</code>；3.10 以下要另裝 <code>tomli</code></td>
      </tr>
      <tr>
          <td>巢狀很深時</td>
          <td>括號多但結構清楚</td>
          <td>縮排表示結構，深的時候最簡潔</td>
          <td>表頭要寫完整路徑，深的時候冗長</td>
      </tr>
  </tbody>
</table>
<p>型別由寫法決定這一項，兩種格式的差別要實測才看得清楚。用 PyYAML 6.0.3（實作 YAML 1.1）的 <code>safe_load</code> 解析下面這幾行：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="nt">country</span><span class="p">:</span><span class="w"> </span><span class="kc">no</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">mode</span><span class="p">:</span><span class="w"> </span><span class="m">010</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w"></span><span class="nt">port_map</span><span class="p">:</span><span class="w"> </span><span class="m">22</span><span class="p">:</span><span class="m">22</span><span class="w">
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="w"></span><span class="nt">version</span><span class="p">:</span><span class="w"> </span><span class="m">1.10</span><span class="w">
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="w"></span><span class="nt">on</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span></span></span></code></pre></div><p>得到的是 <code>{'country': False, 'mode': 8, 'port_map': 1342, 'version': 1.1, True: True}</code>：<code>no</code> 被讀成布林值、<code>010</code> 被讀成八進位的 8、<code>22:22</code> 被讀成六十進位的 1342、<code>1.10</code> 被讀成浮點數 1.1，連鍵名 <code>on</code> 都變成了布林值 <code>True</code>。每一個值都解析成功，沒有任何錯誤訊息；<code>no</code> 被讀成 <code>false</code> 這一種，英文資料常稱為 Norway problem（挪威的國家代碼 <code>NO</code> 變成布林值）。放回這份清單：<code>firewall-cmd --query-service=https</code> 印出的是字串 <code>yes</code>，清單用 YAML 寫成 <code>expect_stdout: yes</code> 時，PyYAML 把值讀成布林值 <code>True</code>，跟輸出的字串比對永遠不相等，這一項每次都報警告；<code>billed_when_stopped: no</code> 也一樣被讀成 <code>False</code>。</p>
<p>同樣五個值寫成 TOML、不加引號，逐行各自交給 Python 的 <code>tomllib</code> 解析，結果是：兩個版本都報錯（<code>no</code>、<code>010</code>）、只有 TOML 1.1 的解析器接受並讀成另一個型別（<code>22:22</code>）、兩個版本都照合法的型別讀（<code>1.10</code> 讀成浮點數、<code>on</code> 是普通的鍵名）。表格的「同左」代表 3.15 的結果跟 3.12 相同：</p>
<table>
  <thead>
      <tr>
          <th>TOML 這一行</th>
          <th>Python 3.12（TOML 1.0）</th>
          <th>Python 3.15（TOML 1.1）</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>country = no</code></td>
          <td><code>TOMLDecodeError: Invalid value</code></td>
          <td>同左</td>
      </tr>
      <tr>
          <td><code>mode = 010</code></td>
          <td><code>TOMLDecodeError: Expected newline or end of document after a statement</code>（整數不允許前導零，解析器讀完 <code>0</code> 就結束這個值）</td>
          <td>同左</td>
      </tr>
      <tr>
          <td><code>port_map = 22:22</code></td>
          <td><code>TOMLDecodeError</code></td>
          <td><code>datetime.time(22, 22)</code></td>
      </tr>
      <tr>
          <td><code>version = 1.10</code></td>
          <td>浮點數 <code>1.1</code></td>
          <td>同左</td>
      </tr>
      <tr>
          <td><code>on = true</code></td>
          <td><code>{'on': True}</code>（<code>on</code> 是普通的鍵名）</td>
          <td>同左</td>
      </tr>
  </tbody>
</table>
<p>TOML 擋下的是不屬於任何型別的寫法（<code>no</code>、<code>010</code>），屬於另一個合法型別的寫法它照樣接受：<code>1.10</code> 是合法的浮點數，所以被讀成 1.1；<code>22:22</code> 在 TOML 1.1 是合法的時間（1.1 允許省略秒），所以 1.1 的解析器把它讀成時間 22:22。差別在範圍：YAML 1.1 的解析器會替幾乎任何不加引號的值找一個型別，TOML 只在寫法剛好合乎某個型別時才這樣做，其餘的拒絕整份檔案。由實測得出這份清單的寫法：版本號、時間字串、連接埠對應這類「看起來像數字或時間、內容是字串」的值，在兩種格式裡都一律加引號，不依賴解析器拒絕它。</p>
<p>YAML 1.2 把布林值限定成 <code>true</code> 與 <code>false</code> 及其大小寫變體，部分隱式型別的問題因此消失，但一份 YAML 檔被讀成哪個版本的規則，由讀它的解析器決定，寫檔的人控制不了。</p>
<p>由比對的結果，這份清單選 TOML：它滿足全部需求，付出的代價是巢狀的子項目要寫完整的表頭（<code>[[entrypoint.verify]]</code>），以及 Python 的標準函式庫只能讀、不能寫，要由程式產生 TOML 時得另裝套件。這個選擇有兩個前提：主機上的 Python 是 3.11 以上，以及清單巢狀不深、由人手動維護。RHEL 9、Debian 11、Ubuntu 22.04、Amazon Linux 2023 內建的 Python 更舊，在那些主機上 TOML 一樣要另裝 <code>tomli</code>：主機上可以裝套件，或可以把 <code>tomli</code> 這個純 Python 的套件跟檢查腳本放在一起部署時，仍然選 TOML；兩者都不行時選 JSON，人要讀的說明另外寫成一個欄位（例如每一筆的 <code>note</code>），讓說明跟資料仍然放在同一份檔案裡。巢狀很深（例如 Kubernetes 的 manifest）或要由程式大量產生時，YAML 或 JSON 會比較合適。</p>
<h2 id="清單用到的-toml-寫法">清單用到的 TOML 寫法</h2>
<p>示範清單的頂層有這幾段：<code>[vars]</code>（主機上的路徑與網域，給其他欄位引用）、<code>[host]</code>（這台主機是什麼）、<code>[listen_audit]</code>（loopback 以外允許接聽的連接埠）、<code>[[entrypoint]]</code>（對外入口與每一層防護）、<code>[[log]]</code>（定期讀的 log 與計數門檻）、<code>[[accumulation]]</code>（會累積的資料與上限）、<code>[[filesystem]]</code>（檔案系統的剩餘空間）、<code>[updates]</code>（自動更新的設定）、<code>[[suspend]]</code> 與 <code>[[resume]]</code>（暫停與恢復上線的步驟）。</p>
<p>清單的每一類項目寫成一個表格陣列（array of tables），每一筆以 <code>[[類別]]</code> 開頭；這裡的表格（table）是 TOML 的一組鍵值，等於程式裡的一個物件，定義見 <a href="/blog/devops/infra/knowledge-cards/toml/" data-link-title="TOML" data-link-desc="讀到 .toml 設定檔、要判斷一個值會被解析成什麼型別、要在 JSON、YAML、TOML 之間選設定檔格式，或解析器版本對不上而報錯時">TOML</a> 知識卡。一筆項目底下又有好幾筆子項目時，用巢狀的表格陣列。下面是示範清單對外入口那一段的節錄，原檔這個入口有五層防護，這裡只留兩層，<code>enforced_by</code> 也照節錄改成兩層：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="p">[[</span><span class="nx">entrypoint</span><span class="p">]]</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="nx">name</span> <span class="p">=</span> <span class="s2">&#34;https&#34;</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="nx">port</span> <span class="p">=</span> <span class="mi">443</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="nx">protocol</span> <span class="p">=</span> <span class="s2">&#34;tcp&#34;</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="nx">exposure</span> <span class="p">=</span> <span class="s2">&#34;cdn-only&#34;</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="nx">where</span> <span class="p">=</span> <span class="s2">&#34;host&#34;</span>
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="nx">enforced_by</span> <span class="p">=</span> <span class="p">[</span><span class="s2">&#34;oci-nsg&#34;</span><span class="p">,</span> <span class="s2">&#34;firewalld&#34;</span><span class="p">]</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl">
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">  <span class="p">[[</span><span class="nx">entrypoint</span><span class="p">.</span><span class="nx">verify</span><span class="p">]]</span>
</span></span><span class="line"><span class="ln">10</span><span class="cl">  <span class="nx">layer</span> <span class="p">=</span> <span class="s2">&#34;oci-nsg&#34;</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl">  <span class="nx">where</span> <span class="p">=</span> <span class="s2">&#34;console&#34;</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl">  <span class="nx">console_path</span> <span class="p">=</span> <span class="s2">&#34;OCI Console → Networking → Virtual cloud networks → &lt;vcn&gt; → Network security groups → &lt;nsg&gt; → Security rules&#34;</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">  <span class="nx">expect</span> <span class="p">=</span> <span class="s2">&#34;TCP 443 的 ingress 規則只有 Cloudflare 的 IPv4 位址段（每段一條），沒有 0.0.0.0/0&#34;</span>
</span></span><span class="line"><span class="ln">14</span><span class="cl">
</span></span><span class="line"><span class="ln">15</span><span class="cl">  <span class="p">[[</span><span class="nx">entrypoint</span><span class="p">.</span><span class="nx">verify</span><span class="p">]]</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl">  <span class="nx">layer</span> <span class="p">=</span> <span class="s2">&#34;firewalld&#34;</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl">  <span class="nx">where</span> <span class="p">=</span> <span class="s2">&#34;host&#34;</span>
</span></span><span class="line"><span class="ln">18</span><span class="cl">  <span class="nx">command</span> <span class="p">=</span> <span class="s2">&#34;firewall-cmd --query-service=https&#34;</span>
</span></span><span class="line"><span class="ln">19</span><span class="cl">  <span class="nx">expect_stdout</span> <span class="p">=</span> <span class="s2">&#34;yes&#34;</span></span></span></code></pre></div><p><code>[[entrypoint]]</code> 每出現一次，就在 <code>entrypoint</code> 陣列裡加一筆；它下面的 <code>[[entrypoint.verify]]</code> 屬於最近的那一筆 <code>entrypoint</code>，所以每個入口各有自己的一組驗證方式。縮排只是給人讀的，TOML 不用縮排表示結構，縮排寫錯不改變解析的結果。兩層都有 <code>where</code>，意思不同：入口那一層的 <code>where = &quot;host&quot;</code> 代表接聽程式在這台主機上，腳本會確認有程序在聽這個連接埠；每一筆 <code>verify</code> 的 <code>where</code> 決定那一層防護由誰檢查。</p>
<p>只有幾個欄位的子項目可以寫成同一行的表格（inline table），例如暫停步驟裡確認資料庫完整的那一項：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="ln">1</span><span class="cl"><span class="nx">verify</span> <span class="p">=</span> <span class="p">{</span> <span class="nx">command</span> <span class="p">=</span> <span class="s2">&#34;sqlite3 -init /dev/null -readonly -safe {{collector_db}} &#39;pragma quick_check&#39;&#34;</span><span class="p">,</span> <span class="nx">expect_stdout</span> <span class="p">=</span> <span class="s2">&#34;ok&#34;</span> <span class="p">}</span></span></span></code></pre></div><p><code>{{collector_db}}</code> 是清單自己的替換語法，檢查腳本在執行前換成 <code>[vars]</code> 段寫的路徑；TOML 本身沒有變數，這兩個大括號在 TOML 看來只是字串裡的普通字元。清單改由 Jinja2 的範本（Ansible 的 <code>template</code> 模組）產生時，這個寫法跟範本自己的替換語法同形，產生的那一步會先處理它，處理方式見 <a href="/blog/devops/operations/09-host-lifecycle/multi-host-manifest/" data-link-title="多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍" data-link-desc="把單台的主機維護清單擴充到多台主機：每台相同、依角色不同、每台不同的值怎麼分層，合併放在部署時產生每台一份、由檢查腳本讀多份、依角色過濾還是執行時覆寫，用 Ansible 的 inventory、群組變數與範本產生 TOML 清單時的變數合併、兼任角色、型別、語法撞名、跳脫與產生時的驗證，由 Terraform 的輸出產生 inventory、terraform plan 能取代哪些主控台上的確認，對多台執行檢查腳本時 exit code 的讀法與執行身分的代價，以及單台清單夠用的條件">多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</a>〈範本的 {{ }} 與清單自己的替換語法同形〉。TOML 1.0 的 inline table 要寫在一行內、結尾不能有逗號；1.1 放寬了這兩條，解析器版本不同時會怎樣，寫在〈TOML 1.0 與 1.1 的差異與解析器的版本〉。</p>
<p>字串有兩種寫法，差在跳脫：雙引號的基本字串（basic string）會處理反斜線跳脫，regex 裡的 <code>\.</code> 要寫成 <code>\\.</code>；單引號的字面字串（literal string）不處理任何跳脫，寫什麼就是什麼。清單裡的 regex 用基本字串寫成 <code>&quot;(example\\.com|www\\.example\\.com)&quot;</code>，解析出來是 <code>(example\.com|www\.example\.com)</code>。檢查腳本另外提供 <code>--set 名稱=值</code> 參數，讓維護的人在執行時覆寫清單 <code>[vars]</code> 段裡已有的值（<code>[vars]</code> 沒有那個名稱時，腳本把它判成清單格式錯）；這個值不經過 TOML 解析，腳本原樣代入，所以反斜線只寫一個。值經過 shell 傳給腳本時，雙引號裡的反斜線會被 shell 處理一次，經 SSH 在遠端執行時，本機與遠端的 shell 可能各處理一次，送進腳本的值會少掉反斜線；經 SSH 傳時，外層用雙引號、值本身用單引號包住，遠端的 shell 就不再處理裡面的反斜線。另一個要注意的是這個值被代入的位置：示範清單把 <code>host_re</code> 嵌進一條更長的 regex 中間（先跳過前面幾個以 tab 分隔的欄位，再比對 <code>Host</code> 那一欄），值本身帶了 <code>^</code>、<code>$</code> 錨點時，錨點出現在 pattern 的中間，一行都比對不到。要覆寫 regex 這類值時，先讀清單裡引用它的那一行，知道它被嵌在哪裡，再決定值要怎麼寫。</p>
<p>數字寫成整數時就是整數：位元組上限寫 <code>limit_bytes = 1073741824</code>，解析出來就是 Python 的 <code>int</code>，檢查腳本不必再轉型。TOML 1.0 允許用底線分隔位數（<code>1_073_741_824</code>），讓人讀得出數量級。</p>
<h2 id="toml-10-與-11-的差異與解析器的版本">TOML 1.0 與 1.1 的差異與解析器的版本</h2>
<p>TOML 1.0.0 在 2021 年 1 月發佈，1.1.0 在 2025 年 12 月發佈。1.1 新增的語法有：inline table 可以跨行並在結尾加逗號、基本字串的 <code>\xHH</code> 與 <code>\e</code> 跳脫、時間可以省略秒。這些語法只有支援 1.1 的解析器讀得懂。Python 的 <code>tomllib</code> 在 3.11 加入時支援 1.0，3.15 起支援 1.1。</p>
<p>這個版本差異在開發機與主機的直譯器版本不同時出現，結果有兩種：</p>
<ul>
<li><strong>解析失敗</strong>：同一行 <code>a = {x = 1, y = 2,}</code>，Python 3.15 解析成 <code>{'a': {'x': 1, 'y': 2}}</code>，Python 3.12 丟出 <code>TOMLDecodeError: Invalid initial character for a key part</code>。開發機是 3.15、主機是 Oracle Linux 10 內建的 Python 3.12 時，清單在開發機上測試通過，部署到主機之後解析失敗，腳本一項都不執行。</li>
<li><strong>同一個值在一個版本報錯、在另一個版本被讀成別的型別</strong>：〈主機維護清單對檔案格式的需求：JSON、YAML、TOML 的比對〉那張逐行解析表裡的 <code>port_map = 22:22</code>，Python 3.12 報錯，3.15 讀成時間。開發機是 3.15 時，作者在開發機上測不到任何錯誤，而 3.15 讀到的時間 22:22 本身也不是作者要的值（要的是字串 <code>&quot;22:22&quot;</code>）；換成兩邊都是 3.15 的環境，這個錯讀就一直不會被發現。</li>
</ul>
<p>由此推出兩條寫法：清單照所有讀者裡最舊的那個解析器支援的版本寫（這次是 1.0），並在檔頭的註解寫明；測試時用那個最舊的版本實際解析一次。示範清單的單元測試另外用 <code>uv run --python 3.12</code> 解析清單，開發機的直譯器版本較新時，這一項會擋下 1.1 的語法。主機上用的是另外部署的 <code>tomli</code> 時，支援 1.0 還是 1.1 由 <code>tomli</code> 的版本決定，跟 Python 的版本無關（用 <code>a = {x = 1, y = 2,}</code> 實測，<code>tomli</code> 2.4.0 接受、2.3.0 拒絕），最舊的解析器就是部署上去的那個 <code>tomli</code>，測試時用同一個版本解析。</p>
<h2 id="清單欄位的設計檢查腳本要讀的與人要讀的">清單欄位的設計：檢查腳本要讀的與人要讀的</h2>
<p>清單的每個欄位，讀者是檢查腳本或人。設計時逐個欄位先決定它的讀者，因為兩種讀者對欄位的要求不同：腳本要的是可以照著執行、可以比對的值；人要的是判斷時需要的脈絡。</p>
<p><strong><code>where</code> 決定一個項目誰能檢查</strong>：<code>&quot;host&quot;</code> 是主機上執行得了的只讀指令，腳本會執行它；<code>&quot;external&quot;</code> 要在主機外的機器上做（例如從開發機直連源站，確認繞過 CDN 時連不進來）；<code>&quot;console&quot;</code> 要在雲端主控台上看（Oracle Cloud 的網路安全群組 Network Security Group，簡稱 NSG，裡的規則、CDN 的 SSL 模式）。<code>external</code> 與 <code>console</code> 兩種，腳本只印出來，標成待人工確認。分出這個欄位，是因為入口的防護有幾層不在主機上（雲端的 NSG、CDN 的 SSL 模式），只從主機裡面量的話，雲端那一層的設定改了也看不出來。</p>
<p><strong><code>enforced_by</code> 與 <code>verify</code> 互相對得上</strong>：每個入口宣告它由哪幾層擋住，每一層至少要有一筆驗證方式，每一筆驗證的 <code>layer</code> 都要在 <code>enforced_by</code> 裡。腳本載入清單時比對兩邊，宣告了某一層而沒有寫怎麼驗證的、或寫了驗證而沒有宣告的，都算清單格式錯；同一層寫兩筆驗證是允許的。這條規則讓「這個入口有幾層防護」從一句宣稱變成每一層都可以查的事實。</p>
<p><strong>探針的欄位</strong>（<code>command</code>、<code>expect_exit</code>、<code>ok_exit</code>、<code>expect_stdout</code>、<code>expect_contains</code>、<code>expect_not_contains</code>）描述一次檢查：執行什麼指令、什麼結果算通過。exit code 怎麼判通過分兩種情形：寫了 <code>expect_exit</code> 時，只有等於它的 exit code 算通過，<code>ok_exit</code> 不作用；沒有寫 <code>expect_exit</code> 時，exit code 落在 <code>ok_exit</code> 列的任何一個值都算通過，<code>ok_exit</code> 也沒有寫時只有 0 算通過。<code>ok_exit</code> 是為了 <code>grep -c</code> 這類指令而設的：找不到時它印 0、exit code 是 1，這是正常的結果，不是失敗。</p>
<p><strong>計數的欄位</strong>（<code>measure</code> 與 <code>warn_above</code>）讓 log 的檢查變成數字的比對：探針的指令執行之後，<code>measure</code> 決定怎麼從輸出取出數字（<code>lines</code> 算非空行數、<code>value</code> 取印出的數值、<code>age_days</code> 算印出的時間距今幾天；輸出是 <code>鍵=值</code> 的形式時，<code>value</code> 與 <code>age_days</code> 取等號後面那一段，<code>systemctl show -p NRestarts</code> 印出的 <code>NRestarts=0</code> 就是這種），取出的數字大於 <code>warn_above</code> 時警告。清單在 log 那一段的註解寫明「門檻是第一次上線時的起點，不是量過的基準」：讀到警告的人要知道，門檻本身可能需要調整。</p>
<p><strong>只印不執行的指令寫在 <code>manual_command</code></strong>：<code>command</code> 一律會被執行，而且要通過白名單；只印給人讀、不執行的指令寫在 <code>manual_command</code>，清理資料的做法寫在 <code>purge_command</code>（只在那一項警告時印在同一行）。<code>manual_command</code> 出現在 <code>[[log]]</code> 段落本身（人讀那份 log 用的指令）、<code>[[suspend]]</code> 與 <code>[[resume]]</code> 的步驟（要人執行的動作），以及 <code>where</code> 不是 <code>&quot;host&quot;</code> 的 <code>[[entrypoint.verify]]</code>（在主機外做的確認）。欄位名分開的理由是讀清單的人只看一行就分得出 <code>command</code> 與 <code>manual_command</code> 會不會被執行：示範清單的第一版裡，會執行的指令與只印的指令都叫 <code>command</code>，一行 <code>command</code> 會不會被執行由它所在的段落決定；在 <code>[[entrypoint.verify]]</code> 這張表裡，同一個欄位名又依那一筆的 <code>where</code> 決定：<code>where = &quot;host&quot;</code> 的會執行，其他的只印。讀的人要記得這些規則才分得出來。改名之後，腳本在只印的位置讀到舊名 <code>command</code> 時判成清單格式錯（exit code 2），沿用第一版清單的人不會讓一行只印的指令被安靜地略過。<code>manual_command</code> 與 <code>purge_command</code> 寫的是清理資料與停止主機這類要人判斷時機的動作，腳本裡沒有執行這兩個欄位的程式碼。名字不帶 <code>manual_</code> 而會被執行的欄位還有兩個：會累積的資料用 <code>oldest_command</code> 查最舊一筆資料的時間，<code>[updates]</code> 用 <code>reboot_check</code> 查要不要重開機，兩者同樣在載入清單時比對白名單。</p>
<p><strong>允許「不知道」的值</strong>：暫停步驟的 <code>billed_when_stopped</code> 欄位，除了 <code>yes</code>、<code>no</code>、<code>n/a</code> 之外，還允許 <code>unknown</code>；清單的註解約定，寫 <code>unknown</code> 時在 <code>billing_basis</code> 寫出要去哪裡查（腳本只檢查這個欄位存在）。只給是與否兩個值時，查不到答案的人只能二選一，寫進清單的就是一個猜測；有 <code>unknown</code> 這個值，查不到的項目留下的是一個待查的標記。這個欄位只描述停止主機這一種暫停範圍，終止主機或全部刪除時各項資源的去留與計費，要照 <a href="/blog/devops/operations/09-host-lifecycle/suspend-and-resume-service/" data-link-title="服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序" data-link-desc="一台主機上的服務要暫停一段時間再恢復：只停服務、停止主機、終止主機保留設定、全部刪除靠 repo 重建各自保留什麼與付出什麼，怎麼依保留的月費、重建的成本與漂移量選擇，暫停期間仍在計費的資源（含在作業系統裡 shutdown 不停止計費），仍被 DNS 與 CDN 指向的結果（Cloudflare 521、522，釋放 IP 之後的懸空 DNS），恢復前要重新確認的漂移，以及依元件依賴排出的恢復順序與驗證">服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序</a>〈暫停期間仍在計費的資源〉逐項決定；要讓清單涵蓋其他範圍，欄位就要依範圍分開寫。</p>
<p><strong>清單沒有存的資料：上一次的結果</strong>。維護時 access log 的計數要跟前一次比，恢復時要跟暫停之前的狀態比，而清單只描述「要檢查什麼、允許的範圍是什麼」，沒有任何欄位存上一次量到的值。這是刻意的：<code>tomllib</code> 只能讀不能寫，清單放在 repo 裡、只在有人改它時才改變，量到的值每次執行都不同，不屬於 repo。上一次的結果由檢查腳本寫進它自己的結果檔，預設是主機上的 <code>/var/lib/host-check/last.json</code>，不在 repo 裡；結果檔的格式與比對寫在〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。結果檔只在同一個階段之間比較（這一次的維護跟上一次的維護比，這一次的恢復跟上一次的恢復比），所以要比暫停前後的入口與計數，暫停之前跑一次維護的檢查、恢復之後再跑一次，恢復後那一次的差異就是跟暫停前的比較。公開 IP、管理者 IP、憑證效期這類不是數字的值不在結果檔裡，要照 <a href="/blog/devops/operations/09-host-lifecycle/suspend-and-resume-service/" data-link-title="服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序" data-link-desc="一台主機上的服務要暫停一段時間再恢復：只停服務、停止主機、終止主機保留設定、全部刪除靠 repo 重建各自保留什麼與付出什麼，怎麼依保留的月費、重建的成本與漂移量選擇，暫停期間仍在計費的資源（含在作業系統裡 shutdown 不停止計費），仍被 DNS 與 CDN 指向的結果（Cloudflare 521、522，釋放 IP 之後的懸空 DNS），恢復前要重新確認的漂移，以及依元件依賴排出的恢復順序與驗證">服務暫停與恢復上線：只停服務、停止主機、終止主機到全部刪除各自保留的資源，暫停期間仍計費與仍被指向的資源，恢復前要查的漂移與恢復的順序</a>〈恢復上線前要重新確認的漂移〉另外記下來。</p>
<h2 id="只讀檢查腳本的白名單輸出exit-code-與結果檔">只讀檢查腳本的白名單、輸出、exit code 與結果檔</h2>
<p>檢查腳本 <code>host-check.py</code> 只用 Python 標準函式庫（<code>tomllib</code>、<code>subprocess</code>、<code>shlex</code> 這類），在主機上以 root 執行，讀清單、執行 <code>where = &quot;host&quot;</code> 的項目、印出報告，並把這次的結果寫進自己的結果檔；每一段程式這樣寫的理由在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</a>。腳本這一層能保證的範圍是不改動受檢查的設定與服務狀態：不改設定檔、不啟停服務、不安裝或移除套件、不刪資料。白名單上的工具自己照常寫的檔案不在這個範圍裡：在 Oracle Linux 10 的容器實測，<code>dnf -q history list</code> 與 <code>dnf needs-restarting -r</code> 每次都寫 <code>/var/log/dnf.log</code>、<code>dnf.librepo.log</code>、<code>dnf.rpm.log</code>、<code>hawkey.log</code> 與 <code>/var/cache/dnf/expired_repos.json</code>，加 <code>-C</code> 時仍寫那四份 log；<code>rpm -q</code> 會更新 rpm 資料庫的 <code>rpmdb.sqlite-shm</code>；<code>sqlite3</code> 讀一個 WAL 模式的資料庫時，資料庫旁邊的 <code>-wal</code> 與 <code>-shm</code> 檔不存在的話（服務停掉之後就是這個狀態）會把它們建出來。要讓這些寫入也不發生，得靠下面講的系統層限制，其中 dnf 要寫的整個 <code>/var/log</code> 與 <code>/var/cache/dnf</code> 是實測之後要刻意放行的兩個目錄。腳本這一層的保證靠的是三件事：</p>
<ul>
<li><strong>不經過 shell</strong>：指令用 <code>shlex</code> 切成參數之後直接執行，沒有管線、轉向與命令替換，清單裡的字串在腳本這一層組不出第二條指令。白名單裡的工具自己仍可能有執行 shell 的選項：<code>rpm</code> 帶 <code>--pipe &lt;指令&gt;</code> 時，會把輸出交給 <code>/bin/sh</code> 執行那條指令，所以這一點要靠白名單的選項規則擋下。</li>
<li><strong>執行任何一條之前先比對白名單</strong>：清單欄位裡的指令，以及腳本自己用清單的值組出來的指令（用 <code>systemctl is-enabled &lt;timer&gt;</code> 與 <code>systemctl is-active &lt;timer&gt;</code> 查自動更新的排程、用 <code>journalctl -u &lt;unit&gt;</code> 查服務的紀錄、用 <code>ss</code> 列接聽的連接埠），都在載入清單時比對一次，執行前再比對一次。載入時有一條不符，整份清單算格式錯，一條指令都不執行：一條不符代表清單裡寫了一條白名單不認得的指令，可能是筆誤，也可能是有人寫進了會改動主機的指令，在分清楚之前，同一份清單的其他項目也不能信。代價是清單裡一個筆誤會讓全部檢查停掉，所以接上排程時要對 exit code 2 告警（見下面 exit code 那一段）。執行前那一次比對是保險，正常情形下載入時已經擋下；萬一它不通過，腳本立刻停下、以 exit code 2 結束，在它之前的指令已經執行過。</li>
<li><strong>清理與暫停的動作只印出來</strong>：它們寫在 <code>manual_command</code>、<code>purge_command</code> 這些給人讀的欄位裡，腳本裡沒有執行這些欄位的程式碼。</li>
</ul>
<p>腳本這一層之外，另一種做法是在系統層限制腳本能碰到的範圍，有兩條路：用 systemd 的 service unit 或 <code>systemd-run</code> 執行腳本，由 systemd 在啟動程序之前套上檔案系統、網路與權限的限制；或不以 root 執行，改用專用帳號。兩條路都在一個以 systemd 257 為 init、用 <code>--privileged --cgroupns=private</code> 執行的 Oracle Linux 10 容器裡實測過。量之前先確認限制已經套上：<code>ProtectSystem=strict</code> 底下 <code>touch /var/x</code> 回 <code>Read-only file system</code>，<code>IPAddressDeny=any</code> 底下 <code>curl</code> 逾時（exit 28）。量測環境本身可能讓限制失效而不報錯：OrbStack 的 Linux machine 是 LXC 容器（<code>systemd-detect-virt</code> 印 <code>lxc</code>），systemd 在裡面沒有套上命名空間，也沒有印出任何錯誤或警告，<code>ProtectSystem=strict</code> 底下照樣寫得進 <code>/var</code>，在那種環境量到的是沒有限制的結果。同一組屬性由 Ansible 以 <code>-b</code> 對多台主機執行、以及不加 <code>-b</code> 時 <code>systemd-run</code> 以 <code>Access denied</code> 失敗的輸出，在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈以 -b 與 systemd-run 執行檢查腳本：每台的輸出與 ansible 的 exit code〉。</p>
<p>systemd 的限制用 <code>systemd-run</code> 套上的寫法如下，各屬性實測的效果列在後面的表裡：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 以 root 執行；腳本與清單的路徑換成主機上實際的位置</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">systemd-run --wait --pipe --collect <span class="se">\
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="se"></span>  -p <span class="nv">ProtectSystem</span><span class="o">=</span>strict -p <span class="s1">&#39;ReadWritePaths=/var/log /var/cache/dnf&#39;</span> <span class="se">\
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="se"></span>  -p <span class="nv">StateDirectory</span><span class="o">=</span>host-check -p <span class="nv">StateDirectoryMode</span><span class="o">=</span><span class="m">0700</span> <span class="se">\
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="se"></span>  -p <span class="nv">NoNewPrivileges</span><span class="o">=</span>yes -p <span class="nv">PrivateTmp</span><span class="o">=</span>yes -p <span class="nv">IPAddressDeny</span><span class="o">=</span>any <span class="se">\
</span></span></span><span class="line"><span class="ln">6</span><span class="cl"><span class="se"></span>  python3 &lt;host-check.py 的路徑&gt;</span></span></code></pre></div><table>
  <thead>
      <tr>
          <th>屬性</th>
          <th>實測的效果</th>
          <th>要另外加的設定</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ProtectSystem=strict</code></td>
          <td>整個檔案系統唯讀。dnf 每次都要寫 log，<code>dnf needs-restarting -r</code> 以 exit 1 結束，錯誤是 <code>Read-only file system: '/var/log/dnf.log'</code>；<code>dnf -C check-update</code> 只用快取時同樣因為寫 log 而失敗</td>
          <td><code>ReadWritePaths=</code> 放行 <code>/var/log</code> 與 <code>/var/cache/dnf</code> 之後 exit 0；只放行 <code>/var/log</code> 時改報 <code>/var/cache/dnf/metadata_lock.pid</code> 寫不進去；<code>/var/lib/dnf</code> 不必可寫</td>
      </tr>
      <tr>
          <td><code>StateDirectory=host-check</code></td>
          <td>結果檔的目錄 <code>/var/lib/host-check</code> 可寫，其餘仍唯讀</td>
          <td>目錄預設建成 0755，而且每次啟動都被改回 0755，腳本建目錄時給的 0700 只在目錄不存在時有效；加 <code>StateDirectoryMode=0700</code> 之後目錄 0700、結果檔 0600</td>
      </tr>
      <tr>
          <td><code>NoNewPrivileges=yes</code>、<code>PrivateTmp=yes</code>、<code>IPAddressDeny=any</code></td>
          <td>腳本不提權、不用 <code>/tmp</code>、不連網，對每一項檢查都沒有看得到的影響</td>
          <td>無</td>
      </tr>
      <tr>
          <td><code>PrivateNetwork=yes</code></td>
          <td><code>ss -tlnp</code> 只印出表頭，看不到主機上任何接聽的程式；四個入口的檢查全部變成 <code>WARN</code>；<code>listen_audit</code> 在一個 socket 都看不到時印 <code>WARN</code>（判斷不了有沒有多開的連接埠），修正之前的腳本判成 <code>PASS</code></td>
          <td>不開這一項；禁止對外連線由 <code>IPAddressDeny=</code> 負責</td>
      </tr>
  </tbody>
</table>
<p>其他工具在唯讀的檔案系統上：<code>rpm -q</code> 照常查得到，<code>rpmdb.sqlite-shm</code> 原本就在；<code>sqlite3 -init /dev/null -readonly -safe</code> 讀一個服務正在寫、<code>-wal</code> 與 <code>-shm</code> 都在的資料庫，照常讀得到。服務停掉之後那兩個檔被收掉，同一條指令以 <code>unable to open database file (14)</code> 失敗，因為它要建那兩個檔而檔案系統是唯讀的；暫停時停掉服務再跑的那一輪檢查就是這個情形，從這個 exit 14 推論那一項會是 <code>WARN</code>（實測只跑了單獨的 <code>sqlite3</code> 指令，沒有在服務停掉時跑整份清單）。沒有套限制時，同一條指令讀得到，代價是在資料庫旁邊建出 <code>-shm</code>（32768 位元組）與 <code>-wal</code>（0 位元組）。dnf 那兩個路徑是單獨對 <code>dnf needs-restarting -r</code> 量的；整份清單以 root 執行時只在沒有 <code>ReadWritePaths=</code> 與 <code>StateDirectoryMode=</code> 的設定下跑過（<code>StateDirectoryMode=</code> 在專用帳號那一組用過），那一次比沒套限制時多出的 <code>WARN</code> 只有 dnf 那一項，結果檔照常寫進 <code>StateDirectory</code>，第二次執行的差異段也照常印出。</p>
<p>接上限制之後，<code>reboot_check</code> 的 exit 1 有兩種成因：dnf 判定要重開機，以及 dnf 自己出錯（寫不了 log），只看 exit code 分不開。腳本因此再看輸出：含 dnf 判定要重開機時印的 <code>Reboot is required</code>，才印「更新之後需要重開機」；沒有這一句時印 <code>WARN</code>，理由是判斷不了要不要重開機，後面附上 dnf 的錯誤訊息（<code>Read-only file system: '/var/log/dnf.log'</code>），讀到這一行就去確認 <code>ReadWritePaths=</code> 有沒有放行 dnf 的兩個路徑。修正之前的腳本只看 exit code，把寫不了 log 印成需要重開機；這個判定與上面 <code>listen_audit</code> 的判定，修正之後都在套了同樣限制的容器裡重跑過，各自印出新的 <code>WARN</code>。</p>
<p>專用帳號實測換回的範圍很小。以 <code>useradd -r</code> 建的帳號、加上 <code>systemd-journal</code> 與 <code>adm</code> 兩個群組執行，root 下 37 項 <code>PASS</code> 的那份清單只剩 29 項，其餘變成 <code>WARN</code>。讀不到的有：firewalld 的查詢（<code>firewall-cmd</code> 以 exit 253 回 <code>NotAuthorizedException</code>）、<code>sshd -T</code>（<code>/etc/ssh/sshd_config</code> 的權限是 0600）、服務的資料庫（所在目錄 0750）、放在 <code>/home/opc</code> 底下的建置產物（家目錄 0700）、nginx 的 log 目錄（0711，進得去、列不出檔名）；audit log 在這個容器裡沒有，沒量到，主機上它是 0600 root，從權限推論同樣要 root。資料庫、nginx 的 log 目錄與建置產物這三項，腳本印成 <code>WARN</code>「讀不到 &lt;目錄&gt;（權限，用 sudo 執行）」，清單把那一項設成路徑不存在也算 <code>PASS</code> 時同樣印 <code>WARN</code>：目錄存在而列不出內容，跟目錄不存在是兩件事。修正之前的腳本把兩者都印成「路徑不存在」，前一種情形因此可能判成 <code>PASS</code>。加了群組換回來的只有讀 journal 的兩項，而且前提是 <code>/var/log/journal</code> 的群組與 ACL 由 systemd-tmpfiles 設好：持久 journal 的目錄由 journald 自己建出時是 root:root、檔案 0640，<code>systemd-journal</code> 群組的 <code>journalctl</code> 回 <code>No journal files were opened due to insufficient permissions.</code>，跑過 <code>systemd-tmpfiles --create --prefix /var/log/journal</code> 之後才讀得到。限定指令的 sudoers 這一條走不通，是從腳本的結構推的，沒有實測：腳本直接執行白名單裡的指令、不經過 sudo，sudoers 只能授權整支 <code>python3 host-check.py</code>，範圍跟用 root 執行相同。</p>
<p>所以實測之後的做法是以 root 執行，外面套上上面那段 <code>systemd-run</code> 的屬性。換到的是：白名單解析錯了，指令照樣寫不進受保護的路徑、連不到外面，sqlite 的 <code>-shm</code> 這類工具自己的寫入也一起擋下；為了 dnf 刻意放行的是整個 <code>/var/log</code> 與 <code>/var/cache/dnf</code> 兩個目錄，nginx 這類其他服務的 log 也在 <code>/var/log</code> 底下，白名單漏掉的指令寫得進這兩個目錄裡的任何檔案（只放行 dnf 那四份 log 檔的寫法沒有實測）。兩層的分工是：白名單擋下清單裡寫錯或寫壞的指令，系統層限制擋下白名單漏掉的寫入；下面記的兩輪缺口每一個都是白名單漏掉的寫入或執行，所以只靠白名單時，保證的強度就是白名單解析的正確度。</p>
<p>系統層限制管的是腳本這個程序自己的寫檔與連線，請別的程序代做的改動不在它的範圍裡。在 Ubuntu 24.04（systemd 255）上，以 <code>systemd-run --wait --pipe -p ProtectSystem=strict -p IPAddressDeny=any</code> 執行 <code>systemctl stop &lt;一個暫時的 unit&gt;</code>，exit 0，那個 unit 被停掉，再加上 <code>PrivateNetwork=yes</code> 結果相同：<code>systemctl</code> 經由 unix socket 請 PID 1 停掉服務，改變狀態的是 PID 1，而 PID 1 不受這個程序的限制。加上 <code>RestrictAddressFamilies=&quot;AF_INET AF_INET6&quot;</code>（不含 unix socket）之後，同一條指令以 exit 1 結束，錯誤是 <code>Failed to connect to bus: Address family not supported by protocol</code>；而 <code>systemctl is-active</code> 這類查詢走的是同一條 unix socket，<code>ss</code> 要用 netlink，拿掉這兩種位址族會讓清單裡這幾類檢查一起失敗，保留它們就擋不住代做的改動（整份清單在這個設定下沒有實測，是從這兩個機制推論）。所以 <code>systemctl stop</code>、<code>firewall-cmd --reload</code> 這類交給 PID 1 或 firewalld 代做的改動，只有白名單擋得住；系統層限制補的是白名單漏掉、由腳本自己動手的寫入與連線。</p>
<p>白名單列的是允許的寫法，不在表上的一律拒絕。另一種寫法是列禁止的寫法：只擋已知會改變狀態的選項，其餘放行。列禁止的寫法要把工具現有的每一個會改變狀態的選項都列進去，漏一個就放行一個，工具日後新增的選項也會被放行；列允許的寫法沒有這個問題，代價是清單要用到新的選項時，得先把它加進白名單。白名單分三層比對：</p>
<ul>
<li><strong>執行檔的路徑</strong>：指令的第一個參數只能是裸的工具名，或 <code>/usr/bin</code>、<code>/usr/sbin</code>、<code>/bin</code>、<code>/sbin</code> 底下的路徑。只比對檔名的話，<code>/tmp/evil/grep</code> 與 <code>./grep</code> 都會被當成 <code>grep</code> 放行，執行的是別的程式。裸的工具名依執行時的 <code>PATH</code> 找檔案，腳本又把呼叫者的環境變數傳給指令，所以腳本要經由 sudo（它的 <code>secure_path</code> 會重設 <code>PATH</code>）或 systemd 的 unit（環境是乾淨的）執行；從一個 <code>PATH</code> 被改過的 root shell 直接執行時，這一層擋不住 <code>PATH</code> 前面的同名檔案。</li>
<li><strong>工具</strong>：只有白名單上的工具可以執行。其中 <code>grep</code>、<code>wc</code>、<code>stat</code>、<code>du</code>、<code>df</code>、<code>test</code>、<code>id</code>、<code>uname</code>、<code>getenforce</code>、<code>sestatus</code>、<code>getsebool</code> 沒有寫檔或執行其他程式的選項（在 Oracle Linux 10 上逐支讀過 <code>--help</code>；<code>df --sync</code> 會先把快取寫回磁碟，不改任何檔案的內容），選項不另外比對；其餘的工具各自有選項的規則。</li>
<li><strong>工具的選項</strong>：有選項規則的工具，各自列出允許的子命令與選項，表外的拒絕；長選項的縮寫（<code>--cursor-fi</code> 這種只寫前幾個字元的寫法）也拒絕，因為 <code>journalctl</code> 這類工具接受沒有歧義的前綴，把 <code>--cursor-fi</code> 當成 <code>--cursor-file</code> 執行，只比對全名的規則會放行它。</li>
</ul>
<p>選項要照工具自己的參數規則切分，比對的結果才可靠。<code>systemctl</code>、<code>journalctl</code>、<code>ss</code> 用 <code>getopt_long</code> 的語法；<code>rpm</code> 用 popt、<code>dnf</code> 與 <code>needs-restarting</code> 用 Python 的 argparse，這三支各自的解析器在下面這幾點的切法跟 <code>getopt_long</code> 相同。腳本對這六支用同一套切分：短選項可以合併（<code>-qa</code>）；需要帶值的選項，值可以直接接在選項後面（<code>-pX</code>），也可以是下一個參數；長選項寫成 <code>--名稱</code> 或 <code>--名稱=值</code>；<code>--</code> 之後全部是位置參數。示範腳本的第一版把第一個不以 <code>-</code> 開頭的參數當成子命令，這套切分規則一條都沒有用上，於是放行了兩種寫法：<code>systemctl -p show stop nginx</code> 被第一版判成子命令是 <code>show</code>，而 systemctl 自己解析時把 <code>show</code> 當成 <code>-p</code> 的值，執行的子命令是 <code>stop</code>；<code>dnf -x history -x list install &lt;套件&gt;</code> 裡，兩個 <code>-x</code> 各把 <code>history</code>、<code>list</code> 當成要排除的套件，第一版看到的是允許的 <code>history list</code>，dnf 執行的是 <code>install</code>。現在的版本先把每個選項連同它的值切出來，再判斷剩下的參數裡哪一個是子命令。其餘的工具各有自己的語法：<code>find</code> 的運算式不是選項語法，腳本逐個參數比對；<code>sshd</code>、<code>openssl</code>、<code>sqlite3</code>、<code>ausearch</code> 用單一連字號的長選項或自己的解析器，腳本逐個參數整字比對；<code>firewall-cmd</code> 逐個參數比對允許的樣式。各工具允許的寫法：</p>
<ul>
<li><strong><code>systemctl</code></strong>：子命令只允許 <code>is-active</code>、<code>is-enabled</code>、<code>is-failed</code>、<code>show</code>、<code>status</code>、<code>cat</code>、<code>list-timers</code>、<code>list-units</code>、<code>list-unit-files</code>；需要帶值的選項只允許 <code>-p</code>／<code>--property</code>、<code>-P</code>、<code>-t</code>／<code>--type</code>、<code>--state</code>、<code>-n</code>／<code>--lines</code>、<code>-o</code>／<code>--output</code>、<code>--timestamp</code>。<code>-H</code>／<code>--host</code>（經由 <code>ssh</code> 在另一台主機上執行）與 <code>-M</code>／<code>--machine</code>（改在容器裡執行）不在表上，<code>--root</code>、<code>--user</code> 也不在。</li>
<li><strong><code>dnf</code> 與 <code>needs-restarting</code></strong>：<code>dnf</code> 只允許 <code>needs-restarting</code> 與 <code>history list</code>、<code>history info</code> 這幾個子命令，子命令之前只接受 <code>-q</code>、<code>-C</code>，子命令之後的選項只有 <code>needs-restarting</code> 的 <code>-r</code>、<code>-s</code>、<code>--reboothint</code>、<code>--services</code>；單獨呼叫的 <code>needs-restarting</code> 也只允許這四個。<code>-x</code>、<code>--setopt</code>、<code>--installroot</code>、<code>-y</code> 不在表上。</li>
<li><strong><code>rpm</code></strong>：第一個參數要是 <code>-q</code> 開頭或 <code>--query</code>（查詢模式），格式只能用 <code>--qf</code>／<code>--queryformat</code> 指定；<code>--pipe</code>、<code>--eval</code>、<code>-D</code>／<code>--define</code>、<code>--macros</code>、<code>--rcfile</code>、<code>--root</code>、<code>--dbpath</code> 不在表上。</li>
<li><strong><code>journalctl</code></strong>：需要帶值的選項只允許 <code>-u</code>、<code>--since</code>、<code>--until</code>、<code>-o</code>、<code>-g</code>、<code>-p</code>、<code>-t</code>，<code>-n</code> 的值可帶可不帶；<code>--cursor-file</code>（把這次讀到的位置標記 cursor 寫進指定的檔案）、<code>--vacuum-*</code>、<code>--rotate</code>、<code>--flush</code>、<code>-f</code> 不在表上。</li>
<li><strong><code>ss</code></strong>：只允許列出連線的選項（<code>-H</code>、<code>-l</code>、<code>-n</code>、<code>-t</code>、<code>-u</code>、<code>-a</code>、<code>-p</code>、<code>-4</code>、<code>-6</code> 這些），加上帶值的 <code>-f</code>／<code>--family</code>、<code>-A</code>／<code>--query</code>；<code>-K</code>（關掉符合條件的連線）、<code>-D</code>、<code>-N</code>（換到另一個網路命名空間）、<code>-F</code>（從檔案讀過濾條件）、<code>-r</code>（反查 DNS，會對外連線）不在表上。</li>
<li><strong><code>find</code></strong>：只允許比對條件（<code>-name</code>、<code>-path</code>、<code>-type</code>、<code>-mtime</code>、<code>-size</code>、<code>-user</code>、<code>-perm</code>、<code>-maxdepth</code> 這些）與 <code>-print</code>、<code>-print0</code>、<code>-printf</code>；<code>-delete</code>、<code>-exec</code>、<code>-execdir</code>、<code>-ok</code>、<code>-okdir</code>（執行指令或刪檔）與 <code>-fprint</code>、<code>-fls</code>（把輸出寫進檔案）不在表上。</li>
<li><strong><code>sshd</code></strong>：參數只能是單獨一個 <code>-T</code>（印出生效中的設定）；<code>-T -E &lt;檔案&gt;</code>（把 log 寫進那個檔案）、<code>-f</code>、<code>-o</code> 都拒絕。</li>
<li><strong><code>firewall-cmd</code>、<code>openssl</code>、<code>ausearch</code></strong>：<code>firewall-cmd</code> 只允許 <code>--query-*</code>、<code>--list-*</code>、<code>--get-*</code>、<code>--info-*</code>、<code>--state</code>，以及指定查詢對象的 <code>--permanent</code>、<code>--zone=</code>；<code>openssl</code> 只允許 <code>x509 -noout</code> 加上讀取憑證欄位的選項；<code>ausearch</code> 只允許篩選紀錄的選項（<code>-m</code>、<code>-k</code>、<code>-ts</code>、<code>-te</code>、<code>-i</code> 這些），<code>--checkpoint</code>（把讀到的位置寫進檔案）與 <code>-if</code>（改讀別的檔案）不在表上。</li>
<li><strong><code>sqlite3</code></strong>：一定要同時帶 <code>-init /dev/null</code>、<code>-readonly</code>、<code>-safe</code>，其餘只允許控制輸出格式與執行方式的選項（<code>-header</code>、<code>-csv</code>、<code>-json</code>、<code>-separator</code>、<code>-bail</code>、<code>-batch</code> 這些）；寫在 SQL 參數裡、以 <code>.</code> 開頭的 dot-command（<code>.shell</code>、<code>.output</code>）也拒絕，不只靠 <code>-safe</code>。理由寫在下面講 <code>sqlite3</code> 的那一段。</li>
</ul>
<p>上面這份白名單是兩次測試修出來的。對第一版的白名單函式逐條測試、再在 Oracle Linux 10 容器裡實際執行，找到的缺口是：<code>systemctl -p</code> 與 <code>dnf -x</code> 這兩種沒有照參數規則切分而放行的寫法；<code>journalctl</code> 的規則是列禁止的寫法，漏了 <code>--cursor-file</code>；<code>rpm</code> 與 <code>sshd</code> 的規則只檢查第一個參數，<code>rpm -q &lt;套件&gt; --pipe &lt;指令&gt;</code> 會以 root 執行 shell 指令，<code>sshd -T -E &lt;檔案&gt;</code> 會建立那個檔案；<code>systemctl -H</code> 會讓 systemctl 執行 <code>ssh</code>，當時只讀了 man page、沒有實跑。修補時在同一種容器裡實測，在 <code>PATH</code> 放一個會記錄參數的假 <code>ssh</code>，<code>systemctl -H &lt;使用者&gt;@&lt;主機&gt; status nginx</code> 讓 systemctl 以 root 執行了 <code>ssh -xT -- &lt;使用者&gt;@&lt;主機&gt; systemd-stdio-bridge</code>；第一版只因為把分開寫的 <code>-H &lt;主機&gt;</code> 裡的 <code>&lt;主機&gt;</code> 當成子命令而拒絕了它，<code>-H&lt;主機&gt;</code> 與 <code>--host=&lt;主機&gt;</code> 都會放行。修補時另外找到三個缺口：<code>rpm</code> 的 <code>--eval &quot;%(&lt;指令&gt;)&quot;</code> 與 <code>-D</code> 會經由巨集執行 shell 指令，實測建立了檔案；執行檔的路徑沒有檢查，就是上面「執行檔的路徑」那一層；<code>sqlite3</code> 啟動時讀的 init 檔會在 <code>-safe</code> 生效之前執行，見下面講 <code>sqlite3</code> 的那一段。</p>
<p>除了執行檔的路徑那一個，這些缺口有同一個成因：寫規則的人以為工具會怎麼解析這串參數，工具實際照它自己的解析規則解析，而且支援執行其他程式與寫檔的功能。一支工具要列出哪些選項規則，看它有沒有這幾類功能：執行 shell 的選項或中介機制（巨集、管線、清單能指定的設定檔）、寫檔的選項或函式、執行其他程式或連到別台主機的選項，以及哪些選項需要帶值；這次選項層的缺口全部落在這幾類裡，有這類功能的工具照樣可以進白名單，選項規則要把那些功能排除在允許的寫法之外。另外兩類功能選項規則管不到，要靠上面講的系統層限制：工具不帶任何選項也會寫的檔案（dnf 的 log 與快取、資料庫的 <code>-shm</code> 檔），以及讀取本身就有副作用的對象（在接了 LDAP 這類目錄服務的主機上，<code>id &lt;使用者&gt;</code> 與 <code>stat</code> 查使用者名稱時會連到目錄伺服器；讀裝置檔、碰到自動掛載點會觸發動作；這兩種是從機制推的，沒有實跑）。</p>
<p>白名單防的是清單的內容，所以清單能指定的設定檔都要擋：<code>rpm</code> 的 <code>--macros</code>、<code>--rcfile</code>，<code>sqlite3</code> 的 <code>-init &lt;檔案&gt;</code>。root 自己擁有的預設設定檔不在白名單的防護範圍內：容器實測，在 <code>/root/.rpmmacros</code> 寫進一個 <code>%(touch &lt;檔案&gt;)</code> 巨集，<code>rpm -q bash</code> 就建出了那個檔案，跟 <code>sqlite3</code> 的 <code>~/.sqliterc</code> 是同一類；<code>openssl</code> 讀的 <code>openssl.cnf</code>、<code>dnf</code> 載入的 plugin 也一樣。能改這些檔案的人已經有 root 權限，用不著經過清單。<code>sqlite3</code> 要求 <code>-init /dev/null</code> 的主要理由是清單本身寫得出 <code>-init &lt;檔案&gt;</code>，順帶蓋過了 <code>~/.sqliterc</code>。執行檔的路徑那一個缺口的成因在工具這一層之前：規則只比對了檔名，沒有比對要執行的是哪一個檔案。示範腳本的修補對照每一個缺口寫了單元測試，在 Python 3.12 與 3.15 上各跑一次，兩個版本都全部通過。</p>
<p><code>sqlite3</code> 執行 shell 與寫檔的途徑有三條：dot-command、SQL 函式與啟動時讀的 init 檔。<code>-readonly</code> 只讓資料庫檔案唯讀，<code>sqlite3</code> 的 dot-command（<code>.shell</code>、<code>.system</code>）照樣能執行 shell 指令，SQL 函式 <code>writefile()</code> 照樣能寫檔案系統；<code>-safe</code> 依 sqlite 的 CLI 文件會關掉除了指定資料庫檔案以外、所有可能改動主機的功能（<code>.shell</code>、<code>.system</code>、<code>.output</code> 這類 dot-command，<code>writefile()</code>、<code>readfile()</code>、<code>edit()</code> 這類 SQL 函式，以及 <code>ATTACH</code>），實測加上它之後 <code>.shell</code> 與 <code>writefile()</code> 都被拒絕。<code>-safe</code> 管不到的是啟動時讀的 init 檔：<code>sqlite3</code> 啟動時先執行 <code>-init &lt;檔案&gt;</code> 指定的檔案，沒有指定時執行執行者家目錄的 <code>~/.sqliterc</code>，兩者都在 <code>-safe</code> 生效之前執行，實測 init 檔裡的 <code>.shell touch &lt;檔案&gt;</code> 照樣建出了檔案。<code>-init /dev/null</code> 會取代 <code>~/.sqliterc</code>，寫在 <code>-readonly -safe</code> 之前或之後都擋得住；把 <code>HOME</code> 設成不存在的目錄擋不住，因為 <code>sqlite3</code> 先從系統的帳號資料庫（<code>/etc/passwd</code> 這一類，經由 <code>getpwuid()</code>）查執行者的家目錄，不看 <code>HOME</code>。白名單因此要求 <code>sqlite3</code> 同時帶 <code>-init /dev/null</code>、<code>-readonly</code> 與 <code>-safe</code>，<code>-init</code> 的值只接受 <code>/dev/null</code>。sqlite 的版本紀錄寫了 <code>--safe</code> 在 3.37.0 加入、3.40.1 修正成確實擋下 <code>writefile()</code> 這類 SQL 函式、3.41.0 再補強它擋下的危險 SQL 函式，所以較舊的主機認得這個選項，某些函式卻可能照樣執行得了：主機上的 sqlite 要是 3.41.0 以上（Debian 12 的 3.40.1 介於兩次修正之間），或在主機上對一個測試用的資料庫實際跑一次 <code>sqlite3 -init /dev/null -readonly -safe &lt;資料庫&gt; &quot;select writefile('/tmp/x','a')&quot;</code>，確認被拒絕（Oracle Linux 10 的 sqlite 3.46.1 符合）。</p>
<p>輸出每項一行 <code>PASS</code>、<code>WARN</code> 或 <code>MANUAL</code>，結尾印計數；exit code 0 代表沒有任何 <code>WARN</code>（只有 <code>PASS</code> 與 <code>MANUAL</code>），1 代表至少一項 <code>WARN</code>，2 代表清單格式錯（命令列參數寫錯、<code>--set</code> 的格式錯也是 2，那時 stderr 印的是用法說明而不是 <code>manifest error:</code>）。把格式錯與檢查失敗分成兩個 exit code，是因為兩者的處置不同：格式錯要改清單，檢查失敗要處理主機。把腳本接上排程或告警時，exit code 不是 0 就要告警：只對 1 告警的話，清單格式錯（2）的那段期間一項檢查都沒有跑，也收不到任何通知。<code>MANUAL</code> 不影響 exit code，所以主機外的項目永遠不會觸發告警；那些項目要另外排進維護時段。它們是有人隨時可能做的改動（在主控台改 NSG、改 CDN 的 SSL 模式），改變是<a href="/blog/backend/knowledge-cards/step-change/" data-link-title="Step Change（步階變化）" data-link-desc="判斷一個指標或一項設定的改變能不能從量測的趨勢預測、要用趨勢外推還是跟基準比對才偵測得到時，區分步階變化與逐步累積的變化">步階變化</a>、沒有變化速度可算，檢查的間隔由能接受那個狀態持續多久決定，會造成對外暴露的要另外排定期比對，見 <a href="/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算</a>〈檢查間隔與哪些項目改成自動處理的推算〉裡「改變是步階變化」的那一類；腳本只把它們印成 <code>MANUAL</code>，比對要由那個排程或維護的人做。</p>
<p>結果檔讓每一次的報告能跟前一次比。腳本用 <code>--phase</code> 指定這次跑的是維護（<code>maintain</code>，預設）、暫停（<code>suspend</code>）還是恢復（<code>resume</code>）的檢查，每次執行結束時，把這一次的結果寫進結果檔裡同一個階段的那一段，其他階段的紀錄不動：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-json" data-lang="json"><span class="line"><span class="ln">1</span><span class="cl"><span class="p">{</span><span class="nt">&#34;format&#34;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl"> <span class="nt">&#34;phases&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;maintain&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;finished_at&#34;</span><span class="p">:</span> <span class="s2">&#34;2026-10-26T09:27:01.975703+00:00&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">3</span><span class="cl">                         <span class="nt">&#34;manifest&#34;</span><span class="p">:</span> <span class="s2">&#34;/path/to/host.toml&#34;</span><span class="p">,</span>
</span></span><span class="line"><span class="ln">4</span><span class="cl">                         <span class="nt">&#34;counts&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;PASS&#34;</span><span class="p">:</span> <span class="mi">41</span><span class="p">,</span> <span class="nt">&#34;WARN&#34;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="nt">&#34;MANUAL&#34;</span><span class="p">:</span> <span class="mi">7</span><span class="p">},</span>
</span></span><span class="line"><span class="ln">5</span><span class="cl">                         <span class="nt">&#34;items&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;filesystem/root&#34;</span><span class="p">:</span> <span class="p">{</span><span class="nt">&#34;status&#34;</span><span class="p">:</span> <span class="s2">&#34;PASS&#34;</span><span class="p">,</span> <span class="nt">&#34;value&#34;</span><span class="p">:</span> <span class="mi">72560254976</span><span class="p">}}}}}</span></span></span></code></pre></div><p>這段 JSON 是節錄：<code>items</code> 只留了一項，<code>manifest</code> 的路徑是佔位的值，<code>counts</code> 借用本節最後那台雲端主機修正之後再跑一次的結果，<code>filesystem/root</code> 的值取自下一段本機容器連跑兩次的第一次；實際的結果檔依鍵名排序、每層縮排兩格，這裡壓成幾行。<code>format</code> 是結果檔格式的版本號，腳本讀到不是 1 的版本時，把它當成壞掉的結果檔處理。<code>items</code> 每一項記下狀態與量到的數字，數字來自有 <code>measure</code> 的探針、會累積的資料的總位元組與檔案系統的可用位元組，其他項目是 <code>null</code>。</p>
<p>報告在每項的結果之後、計數之前，印出跟上一次的差異。第一行寫上一次的時間（<code>since: &lt;上一次的 finished_at&gt;</code>），之後每個不同的項目一行：狀態變了的印 <code>CHANGED &lt;項目&gt; WARN→PASS</code>，數字變了的印 <code>DELTA &lt;項目&gt; 舊值 → 新值（差）</code>，這次新增的與上一次有而這次沒有的項目各印 <code>NEW</code>、<code>GONE</code>。在本機的 Oracle Linux 10 容器裡連跑兩次，根檔案系統的可用空間印成 <code>DELTA filesystem/root 72560254976 → 72541392896（-18862080）</code>，也就是兩次之間少了 18862080 位元組（約 18 MiB）。結果檔裡沒有這個階段的上一次紀錄時（第一次執行、第一次跑這個 <code>--phase</code>、結果檔壞掉），這一段只印 <code>since: 沒有前一次的結果（&lt;路徑&gt;）</code>；兩次完全相同時印 <code>since: &lt;時間&gt;，與前一次相同</code>；帶 <code>--no-state</code> 時不印這一段。</p>
<p>結果檔預設在 <code>/var/lib/host-check/last.json</code>，<code>--state-file &lt;路徑&gt;</code> 換位置、<code>--no-state</code> 不讀也不寫。放在 <code>/var/lib</code> 而不放 <code>/run</code>，是因為 <code>/run</code> 在記憶體裡、重開機就清空，而停止主機再啟動等於重開機，放 <code>/run</code> 就比不到暫停前的紀錄；終止主機而不保留開機磁碟時 <code>/var/lib</code> 也一起消失，終止之前要把 <code>last.json</code> 拷出主機。腳本先把內容寫進同一個目錄裡的暫存檔並寫到磁碟，再用 <code>os.replace</code> 換上正式的檔名：同一個檔案系統裡的換名是一次完成的動作，所以程序在寫到一半時被中斷，正式的檔名仍然指著上一次的完整內容（中斷時可能留下一個暫存檔）。目錄以 0700、檔案以 0600 的權限建立，因為結果檔記著主機的連接埠、log 計數與磁碟用量，只給 root 讀。結果檔不是合法的 JSON、格式版本不認得、讀不到或寫不進去時（例如不是以 root 執行），腳本在 stderr 印以 <code>note:</code> 開頭的說明（讀與寫各一行），壞掉的檔案照常覆寫，exit code 照檢查的結果決定：結果檔是比對用的輔助資料，它出錯不代表主機有問題。</p>
<p>這支腳本先在開發機上的 Oracle Linux 10 容器裡測過，第一次在一台 Oracle Linux 10.2 的雲端主機上執行時，結果是 36 項 <code>PASS</code>、12 項 <code>WARN</code>、7 項 <code>MANUAL</code>。12 項 <code>WARN</code> 依成因分成四種：</p>
<ul>
<li><strong>主機的實際狀態跟清單不符</strong>（8 項）：映像檔預設啟用的 <code>rpcbind</code>（111，TCP 與 UDP）與效能監測套件 PCP 的 <code>pmcd</code>（44321）、<code>pmlogger</code>（4330），在 IPv4 與 IPv6 的所有介面上接聽，它們不在清單預期的入口裡。從開發機對公開 IP 實測，TCP 的三個連接埠都連不上，UDP 111 用 <code>rpcinfo -u</code> 查詢得到逾時，雲端規則與主機防火牆都擋住了它們；開發機上測試用的容器映像檔沒有裝這幾個套件，所以在容器裡沒有出現。</li>
<li><strong>紀錄的保存方式跟預期不同</strong>（1 項）：腳本在 journal 裡找不到自動更新的執行紀錄，因為這台主機的 journal 只存在記憶體裡，在那次執行之前的重開機就清空了；成因與兩種狀態的差別見 <a href="/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算</a>〈會累積的資料：寫入者、清理機制與磁碟寫滿的後果〉。</li>
<li><strong>門檻把預期中的紀錄算進去</strong>（2 項）：error log 裡那一行是 413 防護攔下過大的 body 時記下的正常紀錄，要改的是清單的計數條件。access log 的 <code>Host</code> 那一項有兩個成因疊在一起：部署腳本自己的 <a href="/blog/testing/knowledge-cards/smoke-test/" data-link-title="Smoke Test（冒煙測試）" data-link-desc="部署或重建之後要決定先跑哪一組檢查、冒煙測試全部通過而使用者仍回報錯誤，或要判斷一組部署後檢查涵蓋到哪裡、沒有涵蓋到哪裡時">冒煙測試</a> 送出 <code>Host: 127.0.0.1</code>，讓計數本來就不是零，冒煙測試這個成因要改清單的計數條件（排除主機自己送的請求）；那一次執行又用 <code>--set</code> 傳了一個帶 <code>^</code>、<code>$</code> 錨點的網域 regex，這一項計數的是 <code>Host</code> 不是自己網域的行數，而錨點被嵌進 pattern 的中間之後，沒有一行比對得到自己的網域，於是 log 裡全部 130 行都被算成不是自己網域的請求。錨點造成的那一部分是執行時傳入的值寫錯了，改用清單裡的值就消失。</li>
<li><strong>腳本自己的解析錯誤</strong>（1 項）：systemd 回傳的時間格式（<code>Mon 2026-10-26 07:07:43 GMT</code>）腳本讀不懂。</li>
</ul>
<p>主機的實際狀態與紀錄的保存方式，都是讀文件推不出、要在那台主機上量才知道的事；門檻的計數條件與腳本的解析錯誤，是清單與腳本第一次碰到真實資料時要修正的地方。四種成因的處置都不一樣，讀警告時，成因決定要改的是主機、清單還是腳本。這一次的處置是：部署腳本停用並遮蔽 <code>rpcbind</code> 與 PCP、把 journal 改存到磁碟，清單的兩個計數條件排除主機自己的請求與防護的正常紀錄，腳本補上 systemd 時間格式的解析。修正部署之後在同一台主機上再跑一次，結果是 41 項 <code>PASS</code>、1 項 <code>WARN</code>、7 項 <code>MANUAL</code>；剩下的那一項是自動更新的執行紀錄：自動更新上一次執行時，這台主機的 journal 還只存在記憶體裡，那一次的紀錄在之後的重開機被清掉，要等下一次排程執行才會在磁碟上的 journal 留下紀錄。</p>
]]></content:encoded></item><item><title>多台主機的維護清單：值的分層與合併的位置、用 Ansible 的 inventory 與範本產生每台的清單、由 Terraform 的輸出產生 inventory、跨主機執行的 exit code，以及單台清單的適用範圍</title><link>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/multi-host-manifest/</link><pubDate>Mon, 26 Oct 2026 00:00:00 +0800</pubDate><guid>https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/multi-host-manifest/</guid><description>&lt;p>這篇處理 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本&lt;/a> 那份單台清單要套用到好幾台主機時的做法：清單裡的值依「每台相同、依角色不同、每台不同」分類，以及每台各複製一份時，漏改的那一份為什麼沒有任何訊號；合併這些值的步驟可以放在部署時產生每台一份完整的清單、由檢查腳本自己讀多份檔案合併、一份清單裡的項目依角色過濾、或共用一份清單再由執行時的參數覆寫，各自的代價與適用的條件；用 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">組態管理&lt;/a> 工具 Ansible 的 inventory、&lt;code>group_vars&lt;/code> 與 &lt;code>host_vars&lt;/code>、Jinja2 範本產生每台清單的實測，包括變數合併的優先順序（同層的兩個群組依名稱排序，&lt;code>ansible_group_priority&lt;/code> 要寫在 inventory 檔、寫在 &lt;code>group_vars&lt;/code> 裡沒有效果）、兼任兩種角色的主機、YAML 把 &lt;code>no&lt;/code> 讀成 &lt;code>false&lt;/code>、用 &lt;code>to_json&lt;/code> 把值寫進 TOML 時的跳脫與 BMP 以外字元被拒、範本的 &lt;code>{{ }}&lt;/code> 與清單自己的替換語法同形時的兩種結果，以及在產生的那一步就驗證清單；主機由 Terraform 建立時，由它的輸出產生 inventory、再交給 Ansible 產生清單，以及這個做法對 autoscaling 的主機不成立的原因；清單裡要到雲端主控台確認的防火牆規則，&lt;code>terraform plan&lt;/code> 比對得到與比對不到的範圍；用 Ansible 對多台主機執行檢查腳本時，&lt;code>ansible&lt;/code> 指令的 exit code（2、4）與每台的 &lt;code>rc&lt;/code> 怎麼讀、由 Ansible 執行時換來的安全代價；最後是單台清單夠用的條件與改成分層的時機。&lt;/p>
&lt;p>Ansible 與 Terraform 的指令都在容器裡實跑過（ansible-core 2.17.7、Terraform 1.9.8），產生的清單用 Python 3.12 的 &lt;code>tomllib&lt;/code> 解析過；主機的 IP 用文件保留的位址段（203.0.113.0/24）當佔位值。這篇的範本是簡化過的示範，確認的是清單解析得出預期的值；照檢查腳本實際要求的欄位寫範本、經由 SSH 部署到兩台容器、以 &lt;code>-b&lt;/code> 與 &lt;code>systemd-run&lt;/code> 執行 TOML 那一篇的只讀檢查腳本 &lt;code>host-check.py&lt;/code>，以及跨主機的 exit code，在 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理&lt;/a> 實際走過一次。用 Ansible 管理主機上的套件與服務設定（playbook 與 role）不在這篇的範圍。&lt;/p>
&lt;h2 id="主機變成多台時清單裡的值每台相同依角色不同每台不同">主機變成多台時清單裡的值：每台相同、依角色不同、每台不同&lt;/h2>
&lt;p>單台的清單裡每個值都只描述那一台主機；主機變成多台時，同一份清單的值依它跨主機的變化分成三類：&lt;/p>
&lt;ul>
&lt;li>&lt;strong>每台相同的值&lt;/strong>：檢查項目本身、檢查腳本的白名單、log 的 regex、會累積的資料的檢查方式、暫停與恢復的步驟（TOML 那一篇示範清單的 &lt;code>[[suspend]]&lt;/code> 與 &lt;code>[[resume]]&lt;/code> 段）。這些描述的是「怎麼檢查」，跟主機是哪一台無關。白名單是檢查腳本允許執行的工具與每個工具允許的選項，腳本載入清單時逐條比對，有一條不符就把整份清單判成格式錯、一項都不執行；它的規則與比對方式寫在 TOML 那一篇的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。&lt;/li>
&lt;li>&lt;strong>依角色不同的值&lt;/strong>：主機的角色決定它上面跑什麼服務，也就決定要檢查哪些入口與 log。跑 nginx 的 web 主機要檢查 443 與 80 的入口、讀 nginx 的 access log；跑 PostgreSQL 的資料庫主機要檢查 5432 只開放給管理者的來源（清單裡寫 &lt;code>exposure = &amp;quot;admin-only&amp;quot;&lt;/code>）、讀資料庫的 log。TOML 那一篇示範清單的 &lt;code>[[entrypoint]]&lt;/code>、&lt;code>[[log]]&lt;/code> 與 &lt;code>[listen_audit]&lt;/code> 段（loopback 以外允許接聽的連接埠）都是依角色不同的值。&lt;/li>
&lt;li>&lt;strong>每台不同的值&lt;/strong>：主機名、公開 IP、網域、資料庫檔案的路徑、依這台的磁碟大小訂的剩餘空間門檻。&lt;/li>
&lt;/ul>
&lt;p>同一個值可以跨兩類：先給所有主機一個共用的預設，再由角色或某一台覆寫，〈用 Ansible 的 inventory、群組變數與範本產生每台的清單〉範例裡的剩餘空間門檻就是這樣，&lt;code>all&lt;/code> 給 80、web 角色改成 85、&lt;code>web1&lt;/code> 再改成 90。分類看的是一個值最後在各台之間有沒有不同、在哪一層開始不同。&lt;/p></description><content:encoded><![CDATA[<p>這篇處理 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a> 那份單台清單要套用到好幾台主機時的做法：清單裡的值依「每台相同、依角色不同、每台不同」分類，以及每台各複製一份時，漏改的那一份為什麼沒有任何訊號；合併這些值的步驟可以放在部署時產生每台一份完整的清單、由檢查腳本自己讀多份檔案合併、一份清單裡的項目依角色過濾、或共用一份清單再由執行時的參數覆寫，各自的代價與適用的條件；用 <a href="/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">組態管理</a> 工具 Ansible 的 inventory、<code>group_vars</code> 與 <code>host_vars</code>、Jinja2 範本產生每台清單的實測，包括變數合併的優先順序（同層的兩個群組依名稱排序，<code>ansible_group_priority</code> 要寫在 inventory 檔、寫在 <code>group_vars</code> 裡沒有效果）、兼任兩種角色的主機、YAML 把 <code>no</code> 讀成 <code>false</code>、用 <code>to_json</code> 把值寫進 TOML 時的跳脫與 BMP 以外字元被拒、範本的 <code>{{ }}</code> 與清單自己的替換語法同形時的兩種結果，以及在產生的那一步就驗證清單；主機由 Terraform 建立時，由它的輸出產生 inventory、再交給 Ansible 產生清單，以及這個做法對 autoscaling 的主機不成立的原因；清單裡要到雲端主控台確認的防火牆規則，<code>terraform plan</code> 比對得到與比對不到的範圍；用 Ansible 對多台主機執行檢查腳本時，<code>ansible</code> 指令的 exit code（2、4）與每台的 <code>rc</code> 怎麼讀、由 Ansible 執行時換來的安全代價；最後是單台清單夠用的條件與改成分層的時機。</p>
<p>Ansible 與 Terraform 的指令都在容器裡實跑過（ansible-core 2.17.7、Terraform 1.9.8），產生的清單用 Python 3.12 的 <code>tomllib</code> 解析過；主機的 IP 用文件保留的位址段（203.0.113.0/24）當佔位值。這篇的範本是簡化過的示範，確認的是清單解析得出預期的值；照檢查腳本實際要求的欄位寫範本、經由 SSH 部署到兩台容器、以 <code>-b</code> 與 <code>systemd-run</code> 執行 TOML 那一篇的只讀檢查腳本 <code>host-check.py</code>，以及跨主機的 exit code，在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a> 實際走過一次。用 Ansible 管理主機上的套件與服務設定（playbook 與 role）不在這篇的範圍。</p>
<h2 id="主機變成多台時清單裡的值每台相同依角色不同每台不同">主機變成多台時清單裡的值：每台相同、依角色不同、每台不同</h2>
<p>單台的清單裡每個值都只描述那一台主機；主機變成多台時，同一份清單的值依它跨主機的變化分成三類：</p>
<ul>
<li><strong>每台相同的值</strong>：檢查項目本身、檢查腳本的白名單、log 的 regex、會累積的資料的檢查方式、暫停與恢復的步驟（TOML 那一篇示範清單的 <code>[[suspend]]</code> 與 <code>[[resume]]</code> 段）。這些描述的是「怎麼檢查」，跟主機是哪一台無關。白名單是檢查腳本允許執行的工具與每個工具允許的選項，腳本載入清單時逐條比對，有一條不符就把整份清單判成格式錯、一項都不執行；它的規則與比對方式寫在 TOML 那一篇的〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉。</li>
<li><strong>依角色不同的值</strong>：主機的角色決定它上面跑什麼服務，也就決定要檢查哪些入口與 log。跑 nginx 的 web 主機要檢查 443 與 80 的入口、讀 nginx 的 access log；跑 PostgreSQL 的資料庫主機要檢查 5432 只開放給管理者的來源（清單裡寫 <code>exposure = &quot;admin-only&quot;</code>）、讀資料庫的 log。TOML 那一篇示範清單的 <code>[[entrypoint]]</code>、<code>[[log]]</code> 與 <code>[listen_audit]</code> 段（loopback 以外允許接聽的連接埠）都是依角色不同的值。</li>
<li><strong>每台不同的值</strong>：主機名、公開 IP、網域、資料庫檔案的路徑、依這台的磁碟大小訂的剩餘空間門檻。</li>
</ul>
<p>同一個值可以跨兩類：先給所有主機一個共用的預設，再由角色或某一台覆寫，〈用 Ansible 的 inventory、群組變數與範本產生每台的清單〉範例裡的剩餘空間門檻就是這樣，<code>all</code> 給 80、web 角色改成 85、<code>web1</code> 再改成 90。分類看的是一個值最後在各台之間有沒有不同、在哪一層開始不同。</p>
<p>一台主機屬於哪個角色，寫在主機清冊裡（Ansible 那一節的 inventory），不從主機的現況推。從現況推的做法（偵測到裝了 nginx 就套 web 的檢查）讓新主機不必登記就拿到檢查，代價是檢查項目由被檢查的狀態決定：nginx 被誤刪或沒裝上時，web 的檢查跟著消失，報告全部是 <code>PASS</code>，正好是清單要抓的那種故障看不見。</p>
<p>每台各複製一份完整的清單是最直接的做法，代價落在每台相同的值上：要修改一個共用的檢查（例如收緊白名單的一條規則、修正一條 regex），就要改每一份；漏改一份時，那台主機照舊跑舊版的檢查。沒有另外的一致性檢查時（例如在 repo 的 CI 裡用 <code>tomllib</code> 讀出各份清單、比對共用段落是否相同），報告裡沒有任何訊號顯示它的檢查比別台舊。所以多台主機的清單要讓三類值各自只寫一次，再在某一個步驟把它們合併成每台主機實際讀到的清單。</p>
<h2 id="合併每台相同依角色不同每台不同的值的位置部署時產生檢查腳本讀多份依角色過濾或執行時覆寫">合併每台相同、依角色不同、每台不同的值的位置：部署時產生、檢查腳本讀多份、依角色過濾、或執行時覆寫</h2>
<p>合併放在哪裡，決定了合併的結果有沒有寫成一份檔案、合併由哪個元件負責，以及檢查腳本要不要改：</p>
<ul>
<li><strong>部署時產生每台一份完整的清單</strong>：共用的檢查寫成範本，依角色與主機的值逐台產生一份 <code>host.toml</code>，部署到主機上。主機上的檢查腳本照舊只讀一份檔案，不必改；載入時比對白名單的那一道檢查，比對的是最後實際要執行的清單；主機上那份檔案就是那台主機實際被檢查的內容，要知道一台主機檢查了什麼，讀那一份就夠。產生的工具可以是 Ansible（下一節），也可以是部署端的一支小程式：讀共用、角色、主機三份 TOML，合併成一個字典，再用 TOML 的序列化函式庫寫出（Python 的 <code>tomli_w</code>，不在標準函式庫裡），這樣寫出來的引號與跳脫由函式庫處理，下一節範本的跳脫問題不會出現（這支程式的寫法是推論，沒有實作）。代價有三項。一是多了一個產生的步驟。二是要決定產生出來的檔案放哪裡：不放進 repo 時，repo 裡只有範本與變數，要讀某一台的完整清單得先跑一次產生；放進 repo 時，每次改範本都要重新產生並一起提交，換到的是 code review 時看得到每台清單的 diff，所以審查的人要逐台確認改動範圍時放進 repo，主機多、每次改動都牽動每一份時不放。三是產生出來的清單一樣會變舊：部署時某一台連不上、用 <code>--limit</code> 只部署了一部分、新主機沒跑到部署，那台主機就照舊跑上一版的清單，報告裡仍然看不出它比 repo 舊。要讓這種落後看得見，在產生時把 repo 的 commit hash 寫進清單（例如 <code>[host]</code> 的一個欄位），或定期用 <code>--check --diff</code> 比對主機上的清單與範本算出來的內容（寫法在下一節）。</li>
<li><strong>檢查腳本自己讀多份檔案合併</strong>：主機上放一份共用的清單、一份角色的、一份主機自己的，腳本載入時合併。不必另外的工具，代價移到腳本身上：腳本要定義合併規則，同一個鍵出現在兩份檔案時誰勝出，表格陣列（<code>[[entrypoint]]</code> 這類）是接在後面還是整個取代。合併的結果只存在於腳本的記憶體裡，除非腳本另外把合併後的清單印出或寫進結果檔；沒有這一步時，規則寫錯或讀清單的人記錯規則，沒有一份檔案可以拿來對出一台主機實際檢查了什麼。白名單的比對要排在合併之後：比對排在合併之前時，比對的是合併前的片段，合併之後才組出來的指令、或被後一份檔案覆寫的值，沒有經過比對就被執行。</li>
<li><strong>一份清單，項目依角色過濾</strong>：所有主機讀同一份清單，每個依角色不同的項目標出它屬於哪些角色（例如 <code>roles = [&quot;web&quot;]</code>），腳本只執行符合這台主機角色的項目。只要過濾，不必定義覆寫的語意（誰蓋過誰、陣列接續還是取代），改一個共用檢查只改一份檔案，白名單比對的是整份清單，包括別的角色的項目。代價是腳本要改：TOML 那一篇的 <code>host-check.py</code>（原始碼在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/host-check-script-source/" data-link-title="只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制" data-link-desc="要讀主機維護清單的只讀檢查腳本實際怎麼切分各工具的選項、怎麼驗證清單、怎麼判定 PASS、WARN 與 MANUAL、怎麼寫結果檔，或要照它的做法寫自己的檢查腳本時，查完整的 Python 原始碼與每一段這樣寫的理由">只讀檢查腳本 host-check.py 的原始碼與逐段說明：選項的切分與各工具的白名單規則、清單的載入與驗證、指令的執行、各項檢查、路徑與時間的量測、結果檔與命令列，以及這一版的已知限制</a>）沒有這個功能，要決定角色從哪裡來（寫在主機上的一個檔、或執行時傳入），每台的報告也會多出「略過」的項目。這是設計上的選項，示範腳本沒有實作。</li>
<li><strong>共用一份清單，執行時覆寫</strong>：主機上的清單每台相同，每台不同的值在執行時用參數覆寫。<code>host-check.py</code> 的 <code>--set 名稱=值</code> 取代 <code>[vars]</code> 段裡已經有的名稱（<code>[vars]</code> 沒有那個名稱時，腳本把它判成清單格式錯）；腳本讀清單時，把字串欄位裡的 <code>{{名稱}}</code> 換成 <code>[vars]</code> 的值，所以 <code>[host]</code> 的主機名、網域、路徑這類沒有限定允許值的字串欄位都能寫成 <code>{{名稱}}</code>（<code>exposure</code>、<code>where</code> 這類有允許值的欄位，比對允許值在替換之前，寫成 <code>{{名稱}}</code> 會判成格式錯）、由 <code>--set</code> 給值。數字欄位（連接埠、剩餘空間的門檻）不經過替換，表格陣列裡有幾筆項目、陣列裡有哪些值，也不是一個字串，<code>--set</code> 改不到它們，表達不了「多一筆入口」這種結構上的差異。覆寫的值寫在每台主機執行檢查的排程指令上，要讀一台實際檢查了什麼，得同時讀清單與那台的排程設定。</li>
</ul>
<p>選哪一個位置，看每台不同的值是什麼形狀，以及手上已經有什麼工具：</p>
<ul>
<li><strong>每台不同的只有字串（主機名、網域、路徑），而且所有主機的角色相同</strong>：執行時覆寫最省事，不需要任何新工具，清單不必分層；代價是覆寫的值散在各台的排程裡。</li>
<li><strong>角色不只一種，而每台不同的值只有字串</strong>：依角色過濾加上執行時覆寫就夠，角色決定執行哪些項目，<code>--set</code> 給每台的字串；所有主機讀同一份清單，規則最少，前提是檢查腳本可以改（示範腳本沒有過濾的功能）。</li>
<li><strong>每台不同的值有數字或結構（某一台的門檻、某一台多一個入口）</strong>：過濾與覆寫都表達不了，剩下部署時產生與檢查腳本讀多份合併。已經有產生設定檔的工具（Ansible、Terraform），或檢查腳本是別人維護的、不能改時，用部署時產生，換到的是腳本不必改、每台的完整清單看得到；沒有任何產生工具時，部署時產生要另外付引進工具（學一個工具、多一個步驟與一份 inventory）或在部署端寫一支合併程式的成本，而讀多份合併把這份成本換成在檢查腳本裡寫並維護合併規則，所以檢查腳本可以改、又不想在部署端多一個步驟時，讀多份合併是剩下的那一個。</li>
<li><strong>主機由 Terraform 建立</strong>：它負責的是產生 Ansible 要讀的 inventory，清單仍由 Ansible 依範本產生（做法在〈主機由 Terraform 建立時由它的輸出產生 inventory〉）；Terraform 自己的 <code>templatefile()</code> 也能直接寫出每台的清單，放進主機的 user_data 時由 cloud-init 在第一次開機寫到主機上；代價是它只在建立主機時寫一次，之後改清單時，業者不是要重建主機、就是不再套用新的 user_data，主機上的清單就落後了（從 cloud-init 只在第一次開機執行的機制推論，沒有實測），所以會改的值仍要另一個部署步驟。</li>
</ul>
<h2 id="用-ansible-的-inventory群組變數與範本產生每台的清單">用 Ansible 的 inventory、群組變數與範本產生每台的清單</h2>
<p>Ansible 從 inventory（主機清冊：列出受管主機與分組的檔案）讀出受管的主機與分組，變數依層次放在 <code>group_vars/</code> 與 <code>host_vars/</code> 兩個目錄，範本用 Jinja2 寫（Python 的範本引擎：<code>{{ }}</code> 代入變數，<code>{% %}</code> 寫迴圈與條件這類控制語法，<code>|</code> 把變數交給過濾器（filter）這類轉換函式）；inventory 的分層與合併規則在 <a href="/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">Configuration Management（組態管理）</a> 卡。三類值的對應是：每台相同的值寫在範本本身與 <code>group_vars/all.yml</code>，依角色不同的寫在 <code>group_vars/&lt;角色&gt;.yml</code>，每台不同的寫在 <code>host_vars/&lt;主機&gt;.yml</code>。</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c"># inventory.yml：兩台 web 主機、一台資料庫主機</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="w"></span><span class="nt">all</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="w">  </span><span class="nt">children</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="w">    </span><span class="nt">web</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="w">      </span><span class="nt">hosts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="w">        </span><span class="nt">web1</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="w">        </span><span class="nt">web2</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="w">    </span><span class="nt">db</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="w">      </span><span class="nt">hosts</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="w">        </span><span class="l">db1:</span></span></span></code></pre></div><p>四個變數檔各自是一份檔案，檔名就是它所屬的群組或主機：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># group_vars/all.yml：每台相同的預設值</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">disk_warn_percent</span><span class="p">:</span><span class="w"> </span><span class="m">80</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w"></span><span class="nt">collector_path</span><span class="p">:</span><span class="w"> </span><span class="l">/var/lib/collector/monitor.db</span><span class="w">
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="w"></span><span class="nt">country</span><span class="p">:</span><span class="w"> </span><span class="kc">no</span><span class="w">    </span><span class="c"># 刻意示範：這一行會被讀成布林值 false，見下文</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># group_vars/web.yml：web 角色的值；依角色不同的串列用帶角色名的變數名</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">disk_warn_percent</span><span class="p">:</span><span class="w"> </span><span class="m">85</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w"></span><span class="nt">web_listen_ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="m">22</span><span class="p">,</span><span class="w"> </span><span class="m">80</span><span class="p">,</span><span class="w"> </span><span class="m">443</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="w"></span><span class="nt">web_entrypoints</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="w">  </span>- {<span class="w"> </span><span class="nt">name: https, port: 443, exposure</span><span class="p">:</span><span class="w"> </span><span class="l">cdn-only }</span><span class="w">
</span></span></span><span class="line"><span class="ln">6</span><span class="cl"><span class="w">  </span>- {<span class="w"> </span><span class="nt">name: http, port: 80, exposure</span><span class="p">:</span><span class="w"> </span><span class="l">cdn-only }</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># group_vars/db.yml：資料庫角色的值</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">db_listen_ports</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="m">22</span><span class="p">,</span><span class="w"> </span><span class="m">5432</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="w"></span><span class="nt">db_entrypoints</span><span class="p">:</span><span class="w">
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="w">  </span>- {<span class="w"> </span><span class="nt">name: postgres, port: 5432, exposure</span><span class="p">:</span><span class="w"> </span><span class="l">admin-only }</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-yaml" data-lang="yaml"><span class="line"><span class="ln">1</span><span class="cl"><span class="c"># host_vars/web1.yml：只有 web1 不同的值</span><span class="w">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="w"></span><span class="nt">disk_warn_percent</span><span class="p">:</span><span class="w"> </span><span class="m">90</span></span></span></code></pre></div><p>一台主機最後拿到的值是各層合併的結果，<code>ansible-inventory -i inventory.yml --host &lt;主機&gt;</code> 印出合併之後的值。這次的結果：<code>web1</code> 的 <code>disk_warn_percent</code> 是 90（<code>host_vars</code> 蓋過 web 群組的 85），<code>web2</code> 是 85（web 群組蓋過 <code>all</code> 的 80），<code>db1</code> 是 80（db 群組沒有寫，沿用 <code>all</code>）。<code>country</code> 想寫的是挪威的國碼 <code>no</code>（例如給地理封鎖的設定用），三台讀出來卻都是布林值 <code>false</code>：<code>group_vars</code> 是 YAML，Ansible 照 YAML 1.1 的規則把不加引號的 <code>no</code> 讀成 false（YAML 的隱式型別見 <a href="/blog/devops/infra/knowledge-cards/yaml/" data-link-title="YAML" data-link-desc="讀到 .yaml 或 .yml 設定檔、一個沒加引號的值被程式讀成布林值或別的數字、同一份檔案換一個解析器結果不同，或要在 JSON、YAML、TOML 之間選設定檔格式時">YAML</a> 卡），產生出來的清單拿到 <code>country = false</code>。值要當字串用時寫成 <code>country: &quot;no&quot;</code>。</p>
<p>連接埠與入口的變數名帶著角色名（<code>web_listen_ports</code>、<code>db_listen_ports</code>），是為了一台主機同時屬於兩個角色的情形。兩個群組用同一個變數名時（例如都叫 <code>listen_ports</code>），把同一台主機同時放進 web 與 db 兩個群組實測，它拿到的是 web 的 <code>[22, 80, 443]</code>：同一層的群組依名稱的字母順序合併，排在後面的蓋過前面的，web 排在 db 後面。要指定誰優先，替群組設 <code>ansible_group_priority</code>（預設是 1，數字大的較晚合併而勝出）；這個變數寫在 <code>group_vars/db.yml</code> 裡沒有改變結果，寫在 inventory 檔裡 db 群組的 <code>vars</code> 底下（<code>db: {vars: {ansible_group_priority: 10}, hosts: …}</code>）之後，結果變成 db 的 <code>[22, 5432]</code>。這台主機兩組連接埠都要允許，而合併規則只會挑其中一組，兩個群組都定義的每一個變數都會這樣。在 <code>host_vars</code> 替它寫一份完整的值，等於又做出一份副本：web 角色的連接埠改了，這一份就是漏改而沒有訊號的那一份。所以依角色不同的串列，每個角色用自己的變數名，範本依這台主機所屬的群組（Ansible 的 <code>group_names</code>）取聯集：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="ln">1</span><span class="cl"><span class="cp">{%</span> <span class="k">set</span> <span class="nv">eps</span> <span class="o">=</span> <span class="o">(</span><span class="nv">web_entrypoints</span> <span class="k">if</span> <span class="s1">&#39;web&#39;</span> <span class="k">in</span> <span class="nv">group_names</span> <span class="k">else</span> <span class="o">[])</span> <span class="o">+</span> <span class="o">(</span><span class="nv">db_entrypoints</span> <span class="k">if</span> <span class="s1">&#39;db&#39;</span> <span class="k">in</span> <span class="nv">group_names</span> <span class="k">else</span> <span class="o">[])</span> <span class="cp">%}</span><span class="x">
</span></span></span><span class="line"><span class="ln">2</span><span class="cl"><span class="x"></span><span class="cp">{%</span> <span class="k">for</span> <span class="nv">ep</span> <span class="k">in</span> <span class="nv">eps</span> <span class="cp">%}</span><span class="x">
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="x">[[entrypoint]]
</span></span></span><span class="line"><span class="ln">4</span><span class="cl"><span class="x">name = </span><span class="cp">{{</span> <span class="nv">ep.name</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">5</span><span class="cl"><span class="x">port = </span><span class="cp">{{</span> <span class="nv">ep.port</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">6</span><span class="cl"><span class="x">exposure = </span><span class="cp">{{</span> <span class="nv">ep.exposure</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">7</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">8</span><span class="cl"><span class="x"></span><span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span></span></span></code></pre></div><p>同時屬於 web 與 db 的主機，產生出來的清單有三筆 <code>[[entrypoint]]</code>（https、http、postgres），只屬於 web 的主機有前兩筆，<code>tomllib</code> 讀出來都是表格陣列。每個角色的值仍然只寫在那個角色的變數檔裡一次。連接埠的聯集兩邊都有 22，串接之後要用 <code>unique</code> 去掉重複（寫法在下面的範本節錄）。聯集只適用於串列；兩個角色各自給了一個單一值（例如各自的剩餘空間門檻）時，範本要寫出取法（取嚴的那一個，<code>[web_disk_warn, db_disk_warn] | min</code>），或把那個值放在 <code>host_vars</code>，因為兼任的主機只有一台，它的單一值本來就屬於每台不同的值。入口與 log 這類「一串項目、每項幾個欄位」的值，用 <code>{% for %}</code> 逐欄輸出成表格陣列，不直接 <code>to_json</code> 整串：<code>to_json</code> 輸出的是 JSON 物件，TOML 讀不懂（下面〈值寫進 TOML 的方式〉那幾種結果的最後一項）。</p>
<p>Ansible 的 <code>template</code> 模組依範本把這些值寫進每台的清單。下面是範本 <code>host.toml.j2</code> 其餘段落的節錄（上面的 <code>[[entrypoint]]</code> 迴圈也在同一份範本裡）。段名沿用 TOML 那一篇的示範清單；段裡的鍵分兩類：<code>[host]</code> 的 <code>name</code>、<code>[vars]</code> 的 <code>collector_db</code>、<code>[[suspend]]</code> 的 <code>verify</code>、<code>command</code>、<code>expect_stdout</code>，以及 <code>[[entrypoint]]</code> 的 <code>name</code>、<code>port</code> 與 <code>exposure</code> 是那份清單的欄位（<code>[[entrypoint]]</code> 在那份清單裡另有 <code>protocol</code>、<code>where</code>、<code>enforced_by</code> 與各層的 <code>verify</code>，這裡省略），其餘的（<code>allowed_ports</code>、<code>[[filesystem]]</code> 的 <code>mount</code> 與 <code>warn_percent</code>、<code>[vars]</code> 的 <code>country</code>、<code>[[suspend]]</code> 的 <code>name</code>）是為了這裡的示範取的。所以這份範本示範的是值怎麼代入，不能直接配 <code>host-check.py</code> 使用，要換成那份清單實際的鍵名；那份清單另有必填的段落與欄位（<code>[host]</code> 的 <code>provider</code>、<code>region</code>、<code>role</code>，以及 <code>[[log]]</code>、<code>[[accumulation]]</code>、<code>[updates]</code>、<code>[[suspend]]</code>、<code>[[resume]]</code>），其中 <code>role</code> 是單一字串，兼任兩個角色的主機要把群組名串成一個值。照那份清單的欄位寫成的範本在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈變數檔與範本：檢查腳本實際要求的欄位、兼任兩種角色的主機〉：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-jinja" data-lang="jinja"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="x">[host]
</span></span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="x">name = </span><span class="cp">{{</span> <span class="nv">inventory_hostname</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="x">[listen_audit]
</span></span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="x">allowed_ports = </span><span class="cp">{{</span> <span class="o">((</span><span class="nv">web_listen_ports</span> <span class="k">if</span> <span class="s1">&#39;web&#39;</span> <span class="k">in</span> <span class="nv">group_names</span> <span class="k">else</span> <span class="o">[])</span> <span class="o">+</span> <span class="o">(</span><span class="nv">db_listen_ports</span> <span class="k">if</span> <span class="s1">&#39;db&#39;</span> <span class="k">in</span> <span class="nv">group_names</span> <span class="k">else</span> <span class="o">[]))</span> <span class="o">|</span> <span class="nf">unique</span> <span class="o">|</span> <span class="nf">to_json</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln"> 6</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="x">[[filesystem]]
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="x">mount = &#34;/&#34;
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="x">warn_percent = </span><span class="cp">{{</span> <span class="nv">disk_warn_percent</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="x">[vars]
</span></span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="x">collector_db = </span><span class="cp">{{</span> <span class="nv">collector_path</span> <span class="o">|</span> <span class="nf">to_json</span><span class="o">(</span><span class="nv">ensure_ascii</span><span class="o">=</span><span class="kp">False</span><span class="o">)</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="x">country = </span><span class="cp">{{</span> <span class="nv">country</span> <span class="o">|</span> <span class="nf">to_json</span> <span class="cp">}}</span><span class="x">
</span></span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="x">
</span></span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="x">[[suspend]]
</span></span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="x">name = &#34;verify-db&#34;
</span></span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="x">verify = { command = &#34;sqlite3 -init /dev/null -readonly -safe </span><span class="cp">{%</span> <span class="k">raw</span> <span class="cp">%}</span>{{collector_db}}<span class="cp">{%</span> <span class="k">endraw</span> <span class="cp">%}</span><span class="x"> &#39;pragma quick_check&#39;&#34;, expect_stdout = &#34;ok&#34; }</span></span></span></code></pre></div>




<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 在執行指令的這台機器上，依 inventory 逐台產生清單（out/ 目錄要先存在）</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible all -i inventory.yml -c <span class="nb">local</span> -m template <span class="se">\
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="se"></span>  -a <span class="s2">&#34;src=host.toml.j2 dest=out/{{ inventory_hostname }}.toml&#34;</span></span></span></code></pre></div><p><code>ansible</code> 指令直接執行單一模組（ad-hoc 指令），不必另外寫 playbook。指令裡要依環境換的位置：<code>all</code> 是要產生清單的主機範圍，可以換成一個群組名（<code>web</code>）；<code>src</code> 是範本的檔名；<code>-c local</code> 讓三台的檔案都寫在執行指令的這台機器上，<code>dest</code> 裡的 <code>{{ inventory_hostname }}</code> 由 Ansible 換成每台的主機名，所以每台各得到一份。要直接部署到各台主機時拿掉 <code>-c local</code>，Ansible 改成經由 SSH 連到每台主機；上面手寫的 inventory 只寫了主機名，Ansible 會拿 <code>web1</code> 當成連線的位址，所以每台要加上 <code>ansible_host: &lt;IP 或 DNS 名稱&gt;</code>，或讓 <code>~/.ssh/config</code> 有同名的 <code>Host</code>。同時，<code>dest</code> 改成檢查腳本在主機上讀清單的固定位置（絕對路徑）；保留相對路徑的話，檔案會寫在遠端的工作目錄底下。<code>template</code> 與 <code>copy</code> 不會建立 <code>dest</code> 的上層目錄，目錄不存在時實際執行以 <code>Destination directory … does not exist</code> 失敗，而 <code>--check</code> 的預覽報不出這個錯，所以部署時先用 <code>file</code> 模組建目錄。部署到兩台容器的完整 playbook 在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈部署清單：建目錄、copy 與 template、validate 改用檢查腳本的載入函式、&ndash;check 的預覽〉。</p>
<h3 id="值寫進-toml-的方式直接輸出to_json-與-ensure_ascii">值寫進 TOML 的方式：直接輸出、to_json 與 ensure_ascii</h3>
<p>Jinja2 把變數寫進檔案時，用的是 Python 的寫法，不是 TOML 的寫法；<code>to_json</code> 過濾器改用 JSON 的寫法輸出，而 JSON 的字串、數字、布林值與陣列 TOML 都讀得懂。範本裡的字串、布林值與陣列因此都經過 <code>to_json</code>，<code>warn_percent</code> 與 <code>port</code> 那幾行是整數，Python 與 TOML 的寫法相同，直接輸出也解析得出來。各種輸出方式的實測結果：</p>
<ul>
<li><strong>直接輸出 <code>{{ 變數 }}</code></strong>：布林值寫成 <code>False</code>，<code>tomllib</code> 在那一行報 <code>Invalid value</code>；字串不帶任何引號，一律解析失敗；字串陣列依內容選引號，<code>['nginx']</code> 被 TOML 當成字面字串（literal string）而讀得懂，同時含單引號與雙引號的值解析失敗，含反斜線的值（<code>C:\dir</code>）被寫成兩個反斜線，而字面字串不處理跳脫，解析出來的值安靜地多了一個反斜線。</li>
<li><strong><code>to_json</code></strong>：引號與反斜線被跳脫成 TOML 也接受的形式，<code>a&quot;b\c.example</code> 輸出成 <code>&quot;a\&quot;b\\c.example&quot;</code>，<code>tomllib</code> 解析回來是原本的字串。非 ASCII 字元被寫成反斜線加 <code>u</code> 與四位十六進位碼的跳脫：基本多文種平面（Basic Multilingual Plane，BMP，碼位 U+0000 到 U+FFFF）以內的中文照樣讀得懂；平面以外的字元（emoji、CJK 擴充 B 區的字）在 JSON 的 <code>\u</code> 跳脫裡要拆成兩個 16 位元的碼（surrogate pair），而 TOML 的 <code>\u</code> 只接受一個完整的 Unicode 碼位，拆開的半個碼位被 <code>tomllib</code> 以 <code>Escaped character is not a Unicode scalar value</code> 拒絕。</li>
<li><strong><code>to_json(ensure_ascii=False)</code></strong>：非 ASCII 字元原字照寫，上面幾種字串都解析得出來，範本裡的字串因此都用這個寫法。</li>
<li><strong><code>to_json</code> 遇到 TOML 沒有的值</strong>：變數沒有設值時的 <code>null</code>，與值是字典時的 JSON 物件，<code>tomllib</code> 分別報 <code>Invalid value</code> 與 <code>Expected '='</code>。範本用到的變數都要在變數檔裡設值；字典與字典的串列照上面 <code>[[entrypoint]]</code> 的寫法逐欄輸出。</li>
</ul>
<h3 id="範本的---與清單自己的替換語法同形">範本的 {{ }} 與清單自己的替換語法同形</h3>
<p>示範清單用 <code>{{collector_db}}</code> 表示「執行前換成 <code>[vars]</code> 的 <code>collector_db</code>」，這個寫法跟 Jinja2 的替換語法同形，Ansible 產生清單時會先把它當成 Jinja2 的變數處理，實測有兩種結果：</p>
<ul>
<li><strong>產生失敗</strong>：Ansible 沒有叫 <code>collector_db</code> 的變數時，那一台的產生停下來，訊息是 <code>AnsibleUndefinedVariable: 'collector_db' is undefined</code>。</li>
<li><strong>安靜替換</strong>：Ansible 剛好有一個同名的變數（例如在 <code>host_vars</code> 寫了 <code>collector_db: /srv/other.db</code>）時，產生成功，指令裡的 <code>{{collector_db}}</code> 被換成 <code>/srv/other.db</code>，清單的 <code>[vars]</code> 寫的卻是另一個路徑，報告看不出這兩者不一致。</li>
</ul>
<p>兩種結果在產生的那一刻給的訊號不同：產生失敗讓產生停下來，安靜替換產生出一份看起來正確、卻檢查了另一個檔案的清單，所以安靜替換比較危險。範本裡把那一段包在 <code>{% raw %}</code> 與 <code>{% endraw %}</code> 之間，Jinja2 原樣輸出，產生的清單裡留著 <code>{{collector_db}}</code>，交給檢查腳本去換；上面的變數檔把 Ansible 這一端的變數取名 <code>collector_path</code>，跟清單的 <code>collector_db</code> 分開，也是為了避開同名。另一種避開撞名的做法是讓清單的替換語法換一個寫法（例如 <code>${collector_db}</code>，Jinja2 預設不處理），代價是改了既有清單與檢查腳本的格式。</p>
<h3 id="產生時驗證清單與比對主機上的清單">產生時驗證清單與比對主機上的清單</h3>
<p>上面的錯誤裡，布林值與字串直接輸出、<code>null</code>、字典、BMP 以外字元的跳脫這幾種會產生出解析不了的清單；另外三種產生出的清單解析得了、值卻是錯的（<code>country: no</code> 變成 <code>false</code>、字面字串多一個反斜線、<code>{{collector_db}}</code> 被安靜替換），<code>validate</code> 攔不到它們。解析不了的那幾種沒有攔下來的話，壞掉的清單照樣部署到主機，等下一次排程執行時，檢查腳本才以 exit code 2（清單格式錯）結束，那台主機在修好之前一項檢查都沒有跑。<code>template</code> 模組的 <code>validate</code> 參數在寫出檔案之前，先對暫存檔執行一條指令，指令失敗就不寫出：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># validate 的值是一條指令，%s 由 Ansible 換成暫存檔的路徑；參數用 JSON 形式傳，避免 key=value 寫法把指令裡的空白切開</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible all -i inventory.yml -c <span class="nb">local</span> -m template -a <span class="s1">&#39;{&#34;src&#34;: &#34;host.toml.j2&#34;, &#34;dest&#34;: &#34;out/{{ inventory_hostname }}.toml&#34;, &#34;validate&#34;: &#34;python3 -c \&#34;import tomllib,sys; tomllib.load(open(sys.argv[1],&#39;</span><span class="s2">&#34;&#39;&#34;</span><span class="s1">&#39;rb&#39;</span><span class="s2">&#34;&#39;&#34;</span><span class="s1">&#39;))\&#34; %s&#34;}&#39;</span></span></span></code></pre></div><p>實測的結果：範本產生出 <code>flag = True</code> 這種 TOML 讀不懂的值時，那一台以 <code>failed to validate</code> 失敗，訊息帶著 <code>TOMLDecodeError: Invalid value</code>，目的地沒有寫出任何檔案；範本正常時照常寫出。用 <code>tomllib</code> 當 <code>validate</code> 只驗得到 TOML 的語法；清單的欄位與白名單是否符合，要讓檢查腳本自己讀一次才知道。檢查腳本的 <code>load_manifest()</code> 只載入與驗證清單、不執行任何檢查，用 Python 的 <code>runpy</code> 載入腳本再呼叫它當 <code>validate</code> 時，<code>exposure</code> 寫了不在允許值裡的值那一台以 <code>failed to validate</code> 失敗，主機上的清單維持舊版；<code>validate</code> 的完整寫法與載入方式的取捨在 <a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈部署清單：建目錄、copy 與 template、validate 改用檢查腳本的載入函式、&ndash;check 的預覽〉。</p>
<p>主機上的清單跟範本算出來的內容是否一致，用 <code>--check --diff</code> 比對：<code>ansible &lt;主機&gt; -i inventory.yml -m template -a &quot;src=host.toml.j2 dest=&lt;清單的路徑&gt;&quot; --check --diff</code> 不寫任何檔案，只印出兩者的差異。實測時先把產生好的檔案手改成別的內容，這條指令印出 <code>--- before</code> 與 <code>+++ after</code> 兩邊的差異行、把那一台標成 <code>CHANGED</code>，檔案維持手改後的內容。定期對所有主機跑一次，標成 <code>CHANGED</code> 的那幾台，就是清單落後於 repo、或在主機上被改過的主機。這條指令有主機標成 <code>CHANGED</code> 時，<code>ansible</code> 的 exit code 仍是 0，所以接上排程時要讀輸出裡的 <code>CHANGED</code>，不能只看 exit code（對兩台容器經由 SSH 實測，在其中一台手改清單之後）。</p>
<h2 id="主機由-terraform-建立時由它的輸出產生-inventory">主機由 Terraform 建立時由它的輸出產生 inventory</h2>
<p>主機由 Terraform 建立、而且每台主機各自是 Terraform 的一個資源時，它的 <a href="/blog/devops/infra/knowledge-cards/state/" data-link-title="State（IaC 狀態檔）" data-link-desc="IaC 工具用來記錄每個納管資源在雲端真實樣貌的快照，是比對差異與排定操作順序的依據">State（IaC 狀態檔）</a> 已經記著每台主機的 IP；inventory 另外手寫一份的話，新增或刪除主機時要改兩個地方，漏改的那一份就指著不存在的主機，或漏掉新的主機。由 Terraform 的輸出產生 inventory，主機清冊就只剩一個來源，Ansible 再讀這份 inventory 產生每台的清單。下面的設定用 <code>locals</code>（設定檔內部的具名常數）代替實際的主機資源（實際的設定裡，IP 來自主機資源的屬性，例如 Oracle Cloud 的 <code>oci_core_instance</code> 的 <code>public_ip</code>），每台主機的角色寫成串列，用 <code>local_file</code> 寫出 inventory：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-hcl" data-lang="hcl"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="k">locals</span> {
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="n">  hosts</span> <span class="o">=</span> {
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="n">    web1</span>  <span class="o">=</span><span class="n"> { roles</span> <span class="o">=</span><span class="n"> [&#34;web&#34;], public_ip</span> <span class="o">=</span> <span class="s2">&#34;203.0.113.10&#34;</span> }
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="n">    db1</span>   <span class="o">=</span><span class="n"> { roles</span> <span class="o">=</span><span class="n"> [&#34;db&#34;], public_ip</span> <span class="o">=</span> <span class="s2">&#34;203.0.113.20&#34;</span> }
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="n">    both1</span> <span class="o">=</span><span class="n"> { roles</span> <span class="o">=</span><span class="n"> [&#34;web&#34;, &#34;db&#34;], public_ip</span> <span class="o">=</span> <span class="s2">&#34;203.0.113.30&#34;</span> }
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">  }
</span></span><span class="line"><span class="ln"> 7</span><span class="cl">}<span class="c1">
</span></span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="c1">
</span></span></span><span class="line"><span class="ln"> 9</span><span class="cl"><span class="c1"># 依角色分組，寫出 Ansible 讀得懂的 inventory.yml
</span></span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="c1"></span><span class="k">resource</span> <span class="s2">&#34;local_file&#34; &#34;inventory&#34;</span> {
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="n">  filename</span>        <span class="o">=</span> <span class="s2">&#34;${path.module}/inventory.yml&#34;</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="n">  file_permission</span> <span class="o">=</span> <span class="s2">&#34;0644&#34; # 預設值是 &#34;0777&#34;</span><span class="err">，</span><span class="k">實際權限再扣掉建立檔案那個程序的</span> <span class="k">umask</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl"><span class="n">  content</span> <span class="o">=</span> <span class="k">yamlencode</span><span class="p">(</span>{
</span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="n">    all</span> <span class="o">=</span> {
</span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="n">      children</span> <span class="o">=</span> {
</span></span><span class="line"><span class="ln">16</span><span class="cl">        <span class="k">for</span> <span class="k">role</span> <span class="k">in</span> <span class="k">distinct</span><span class="p">(</span><span class="k">flatten</span><span class="p">([</span><span class="k">for</span> <span class="k">h</span> <span class="k">in</span> <span class="k">values</span><span class="p">(</span><span class="k">local</span><span class="p">.</span><span class="k">hosts</span><span class="p">)</span> <span class="err">:</span> <span class="k">h</span><span class="p">.</span><span class="k">roles</span><span class="p">]))</span> <span class="err">:</span>
</span></span><span class="line"><span class="ln">17</span><span class="cl"><span class="n">        role</span> <span class="o">=</span><span class="n">&gt; { hosts</span> <span class="o">=</span><span class="n"> { for name, h in local.hosts : name</span> <span class="o">=</span><span class="n">&gt; { ansible_host</span> <span class="o">=</span> <span class="k">h</span><span class="p">.</span><span class="k">public_ip</span> } <span class="k">if</span> <span class="k">contains</span><span class="p">(</span><span class="k">h</span><span class="p">.</span><span class="k">roles</span><span class="p">,</span> <span class="k">role</span><span class="p">)</span> } }
</span></span><span class="line"><span class="ln">18</span><span class="cl">      }
</span></span><span class="line"><span class="ln">19</span><span class="cl">    }
</span></span><span class="line"><span class="ln">20</span><span class="cl">  }<span class="p">)</span>
</span></span><span class="line"><span class="ln">21</span><span class="cl">}
</span></span><span class="line"><span class="ln">22</span><span class="cl">
</span></span><span class="line"><span class="ln">23</span><span class="cl"><span class="n">output &#34;hosts&#34; { value</span> <span class="o">=</span> <span class="k">local</span><span class="p">.</span><span class="k">hosts</span> }</span></span></code></pre></div><p><code>yamlencode</code> 那一段的兩層 <code>for</code>：外層把每台主機的角色串列攤平、取出不重複的角色（<code>web</code>、<code>db</code>），每個角色成為 <code>children</code> 底下的一個群組；內層把角色串列裡含有這個角色的主機收進那個群組的 <code>hosts</code>，每台帶上自己的 <code>ansible_host</code>（Ansible 連線時用的位址）。<code>terraform apply</code> 之後，<code>inventory.yml</code> 裡 <code>both1</code> 同時出現在 db 與 web 兩個群組，上一節的範本就會替它取兩個角色的聯集。角色寫成單一字串時（<code>role = &quot;web&quot;</code>），一台主機只會進一個群組，兼任的主機拿不到另一個群組的任何變數，那個角色的入口與 log 整組不在它的清單裡，而且不會報錯。</p>
<p><code>ansible-inventory -i inventory.yml --graph</code> 列出的分組跟 <code>locals</code> 一致；<code>terraform output -json hosts</code> 印出同一份資料的 JSON，給 Ansible 以外的工具讀。這份 inventory 是 Terraform 管理的一個資源（<code>local_file</code>），它的內容記在 State 裡：手動在檔案末端加一行之後，<code>terraform plan</code> 回報 <code>local_file.inventory will be created</code>，下一次 <code>apply</code> 會把檔案重新寫回 Terraform 算出的內容，手加的那一行消失。plan 寫的是 <code>will be created</code> 而不是修改：<code>local_file</code> 讀取時發現檔案內容跟 State 記的不同，就把這個資源當成不存在，計畫裡改成重新建立（這是從 plan 的訊息推論的機制）。所以每台不同的值裡，Terraform 知道的那幾個（IP、角色）由它產生，其他的（資料庫路徑、門檻）寫在 Ansible 的 <code>host_vars</code>，兩邊各管各的，不在產生出來的檔案上手改；仍有變數要靠 <code>ansible_group_priority</code> 決定哪個群組勝出時，inventory 改由 Terraform 產生之後，它要寫進 <code>yamlencode</code> 裡那個群組的 <code>vars</code>；兼任兩種角色的主機照 Ansible 那一節的聯集寫法，不靠優先順序。</p>
<p>這個做法的前提是每台主機都在 State 裡。由 autoscaling 群組或 instance pool 依流量開關的主機，Terraform 只宣告群組本身，個別主機由業者的服務建立與刪除，不在 State 裡，從 State 產生的 inventory 是空的、或一直落後於實際的主機。這一類主機多半是同一個角色、彼此可以互相取代，每台不同的值只剩主機名這類字串，比較貼近〈合併每台相同、依角色不同、每台不同的值的位置〉的執行時覆寫：同一份清單放進主機的映像檔，或在開機時寫入，主機名由執行時的參數給（這是從機制推出的做法，沒有實測）。這類主機生命週期短，結果檔裡常常沒有上一次的紀錄可比，跨主機的比較要交給監控。</p>
<p><code>local_file</code> 寫出的 inventory 只存在執行 <code>apply</code> 的那台機器上。團隊共用 remote backend 的 State、或在 CI 上跑 <code>plan</code> 時，那台機器的工作目錄裡沒有這個檔（或內容不同），每一次 <code>plan</code> 都回報 <code>local_file.inventory will be created</code>；〈清單裡要到主控台確認的項目與 terraform plan 的比對範圍〉用 <code>plan</code> 偵測防火牆規則的漂移時，同一份 State 的 <code>plan</code> 因此永遠不是 <code>No changes</code>，漂移告警不是一直響、就是被忽略。這種情形下改用直接讀 State 的 inventory plugin（Ansible 的 <code>cloud.terraform</code> collection 的 <code>terraform_state</code>），inventory 不寫成檔案，每次執行時從 State 讀出主機（依那個 collection 的文件推論，沒有實測）；<code>local_file</code> 適合只有一個人、在同一台機器上 <code>apply</code> 與執行 Ansible 的情形。</p>
<h2 id="清單裡要到主控台確認的項目與-terraform-plan-的比對範圍">清單裡要到主控台確認的項目與 terraform plan 的比對範圍</h2>
<p>TOML 那一篇的清單裡，<code>where = &quot;console&quot;</code> 的項目（雲端的網路安全群組（Network Security Group，NSG）規則，概念見 <a href="/blog/devops/infra/knowledge-cards/security-group/" data-link-title="Security Group" data-link-desc="掛在資源網卡層級的有狀態防火牆，逐埠決定哪些來源能連進這個資源">Security Group</a>）只印出來、等人到主控台確認。主機與防火牆規則都由 Terraform 建立時，那些規則的宣告就在 Terraform 的設定裡，<a href="/blog/devops/infra/knowledge-cards/terraform-plan-apply/" data-link-title="terraform plan / apply" data-link-desc="IaC 的兩個核心操作：plan 只看不動（產出差異報告）、apply 真的動（執行差異）"><code>terraform plan</code></a> 會比對 State 裡每個資源的屬性跟雲端的實際值，有人在主控台改了某一條由 Terraform 建立的規則，plan 會列出差異（<a href="/blog/devops/infra/knowledge-cards/drift/" data-link-title="Drift（設定漂移）" data-link-desc="IaC 的 state 與雲端實際狀態之間的不一致，通常因為有人繞過 IaC 直接在 Console 改設定">Drift（設定漂移）</a>）。plan 看不到的是 State 裡沒有的資源：規則在 Terraform 裡寫成各自獨立的資源時（Oracle Cloud 的 <code>oci_core_network_security_group_security_rule</code> 是一條規則一個資源），有人在主控台另外新增一條規則，那一條不在 State 裡，plan 照樣回報 <code>No changes</code>。所以 plan 能取代的是確認既有的規則沒有被改，確認沒有多出來的規則仍然要列出雲端上的規則跟宣告比對。這是從 plan 的比對機制推出的做法，這次沒有對雲端的防火牆規則實測。</p>
<h2 id="用-ansible-對多台主機執行檢查腳本整批的-exit-code每台的-rc-與執行身分的代價">用 Ansible 對多台主機執行檢查腳本：整批的 exit code、每台的 rc 與執行身分的代價</h2>
<p>清單部署到各台之後，檢查腳本可以照舊由每台自己的排程執行，也可以由 Ansible 對整個 inventory 一次執行。exit code 的讀法用一條會依主機名回不同 exit code 的指令（web1 回 0、web2 回 1、db1 回 2）在容器裡實測：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 實測用的指令：-c local 在執行指令的這台機器上跑，-o 讓每台的結果印成一行</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible all -i inventory.yml -c <span class="nb">local</span> -m <span class="nb">command</span> <span class="se">\
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="se"></span>  -a <span class="s2">&#34;sh -c &#39;case {{ inventory_hostname }} in web2) exit 1;; db1) exit 2;; esac&#39;&#34;</span> -o</span></span></code></pre></div><p>輸出裡 web1 標成 <code>CHANGED</code>，web2 與 db1 標成 <code>FAILED!</code>，各自的 <code>&quot;rc&quot;</code> 欄位是 1 與 2；<code>ansible</code> 指令本身的 exit code 是 2，依 Ansible 的文件，這個值代表至少一台主機的任務失敗，跟失敗的是哪一台、各自回了什麼 <code>rc</code> 無關；有主機連不上時這個值是 4（停掉一台容器實測）。所以接上排程時，告警的條件寫成 <code>ansible</code> 的 exit code 不是 0，再讀每台的 <code>rc</code> 決定處置；連不上的主機標成 <code>UNREACHABLE!</code>、沒有 <code>rc</code>，處置是查連線與 SSH，跟清單或主機上的檢查無關。換成檢查腳本時，它的 exit code 1（至少一項 <code>WARN</code>）與 2（清單格式錯）在 Ansible 的輸出裡同樣標成 <code>FAILED!</code>，兩者的處置不同（處理主機或改清單），要讀每台的 <code>rc</code> 才分得開；<code>command</code> 模組每次執行都回報 <code>CHANGED</code>，這個字不代表主機被改動，只代表指令執行了。</p>
<p>換成檢查腳本時，執行的身分與限制要跟單台時相同：TOML 那一篇的腳本以 root 執行，並套上 <code>systemd-run</code> 的檔案系統與網路限制。Ansible 經由 SSH 登入的帳號不是 root，要加 <code>--become</code>（<code>-b</code>）讓 Ansible 在主機上經由 sudo 切成 root：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="ln">1</span><span class="cl"><span class="c1"># 部署後對每台執行檢查；systemd-run 的屬性照 TOML 那一篇〈只讀檢查腳本的白名單、輸出、exit code 與結果檔〉</span>
</span></span><span class="line"><span class="ln">2</span><span class="cl">ansible all -i inventory.yml -b -m <span class="nb">command</span> <span class="se">\
</span></span></span><span class="line"><span class="ln">3</span><span class="cl"><span class="se"></span>  -a <span class="s2">&#34;systemd-run --wait --pipe --collect -p ProtectSystem=strict ... python3 &lt;host-check.py 的路徑&gt; --manifest &lt;清單在主機上的路徑&gt;&#34;</span> -o</span></span></code></pre></div><p><code>...</code> 是 TOML 那一篇那段 <code>systemd-run</code> 的其他屬性，<code>&lt;host-check.py 的路徑&gt;</code> 與 <code>&lt;清單在主機上的路徑&gt;</code> 換成部署時放檔案的位置。不加 <code>-b</code> 時，這條指令裡的 <code>systemd-run</code> 以登入帳號執行，以 <code>Access denied</code> 失敗、<code>rc</code> 是 1，跟檢查腳本至少一項 <code>WARN</code> 的 <code>rc</code> 相同；兩者要看輸出裡有沒有檢查腳本結尾的 <code>summary:</code> 那一行才分得開（<a href="/blog/devops/operations/09-host-lifecycle/hands-on/multi-host-manifest-walkthrough/" data-link-title="多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理" data-link-desc="要照著部署多台主機的維護清單並執行檢查腳本，或要讀跨主機執行時 ansible 的 exit code 與每台的 rc、門檻造成的 WARN、清單格式錯與連不上的主機各自印出什麼時，或要確認 Terraform 寫出的 inventory 的檔案權限、`--check` 預覽與實際部署結果不同的情形時，查在可拋棄的容器上實際跑過的指令、輸出與遇到的錯誤">多台主機維護清單的實作紀錄：在容器上安裝 ansible-core 與 Terraform、由 Terraform 產生 inventory、用 Ansible 的範本部署每台的清單、以 sudo 與 systemd-run 執行檢查腳本，以及清理</a>〈以 -b 與 systemd-run 執行檢查腳本：每台的輸出與 ansible 的 exit code〉）。拿掉 <code>systemd-run</code>、以登入帳號直接執行腳本時，腳本讀不到 firewalld、sshd 設定與幾個目錄：那一篇用加了 <code>systemd-journal</code> 與 <code>adm</code> 群組的專用帳號實測，<code>PASS</code> 的項目從 37 項掉到 29 項；沒有這兩個群組的登入帳號另外讀不到 journal（實作紀錄的同一節）。sudo 要不要密碼依主機的 sudoers 設定而定，要密碼時的兩種給法（<code>-K</code> 在執行時輸入、放進 Ansible Vault 加密的變數）在實作紀錄〈受管主機與控制端的容器：Oracle Linux 10 加 systemd 與 sshd、python:3.12-slim 加 ansible-core 與 Terraform〉。</p>
<p>由 Ansible 加 <code>-b</code> 執行的代價是安全上的：執行 Ansible 的那台機器（控制端）要握有能 SSH 登入每台主機、再經由 sudo 切成 root 的憑證。單台時，檢查是主機自己的 timer 以 root 在本機執行，沒有任何一個地方拿得到所有主機的 root；改由 Ansible 集中執行之後，控制端被入侵就等於每台主機都被入侵，檢查腳本「不改動主機」的保證管不到這一條路。縮小這個範圍的做法有三種，都沒有實測：</p>
<ul>
<li><strong>每台照舊用自己的 timer 執行，集中的只有結果</strong>：結果檔的目錄是 <code>StateDirectoryMode=0700</code>、擁有者是 root，不帶 <code>-b</code> 的帳號讀不到；要讓控制端用一般帳號讀，得把那個目錄改成 0750 並給一個讀取用的群組，或由監控在主機上收集結果再送出去。</li>
<li><strong>sudoers 只允許那一條檢查指令</strong>：Ansible 的 become 會把指令包成 <code>/bin/sh -c '…'</code> 再交給 sudo，限定指令的 sudoers 對不上，所以執行檢查時不加 <code>-b</code>，在 <code>command</code> 模組裡自己寫 <code>sudo -n systemd-run …</code>；部署用另一個帳號與另一把金鑰，只在部署時使用。排程用的那把金鑰被偷時，拿到的只是以 root 執行檢查（從 become 的機制推論）。</li>
<li><strong>ansible-pull</strong>：每台定時從 repo 拉 playbook，在本機產生自己的清單並執行檢查，沒有任何一個地方握有所有主機的 root；代價是每台主機要裝 Ansible、要有讀 repo 的憑證，而且主機要知道自己屬於哪些角色。</li>
</ul>
<p>每台主機的結果檔照舊留在那台主機上（<code>/var/lib/host-check/last.json</code>），每台跟自己的上一次比較；腳本不比較同一個角色的不同主機（web1 的 access log 計數遠高於 web2 這類），這種跨主機的比較要把每台的數字收集到一個地方，那是監控的範圍，各主機的指標怎麼收集見 <a href="/blog/backend/04-observability/vendors/prometheus/multi-host-scraping/" data-link-title="多台主機的指標收集：讀取架構、主機標籤、目標清單、監控主機的位置與主機消失的偵測" data-link-desc="服務分到多台主機之後，Prometheus 怎麼讀到每一台：exporter 的部署位置與連接埠、中央直接讀取與各主機先收再推送的架構、代表主機的 host 標籤加在哪裡、目標清單由什麼產生，主機停掉與主機從清單上消失各自怎麼偵測，以及監控元件放在哪一台、compose 檔怎麼依主機角色拆開；附在單機上模擬第二台主機的驗證">多台主機的指標收集：讀取架構、主機標籤、目標清單、監控主機的位置與主機消失的偵測</a>。</p>
<h2 id="單台清單夠用的條件與改成分層的時機">單台清單夠用的條件與改成分層的時機</h2>
<p>判斷要不要分層，數的是 repo 裡內容不同的清單有幾份，不是主機有幾台：二十台主機都是同一個角色、每台不同的只有主機名時，一份清單部署到每台、主機名用執行時覆寫，repo 裡只有一份，不需要分層；二十台裡只要有一台的門檻不同，內容不同的清單就變成兩份以上，回到下面的判斷。</p>
<p>內容不同的清單有好幾份時，漏改的代價由兩件事決定，兩件事要分開看：</p>
<ul>
<li><strong>漏改的次數</strong>：一年裡累計要改的份數（一次修改要改的份數乘上一年改幾次），每一份都是一次可能漏改的機會。兩份清單、每週改一次，一年要改一百多份；十份清單、一年改一次，一年改十份。</li>
<li><strong>漏改之後持續多久</strong>：漏改的那一份不會讓任何檢查失敗，只會讓那台主機繼續跑舊的檢查，直到有人下一次修改它、或一致性檢查比對到它為止。十份清單、一年改一次時，次數少，可是漏改一份就有一台主機安靜地跑舊檢查到下一年。</li>
</ul>
<p>改成分層的查得到的時機是下面兩者之一：一次修改要改的份數，多到改的人沒辦法在同一次提交裡逐份確認每一份都改了（要另外記「哪幾台已經改了」）；或 repo 已經需要一致性檢查（CI 裡比對各份清單共用段落的那一道）才知道有沒有漏改。兩份清單、每週改一次，每次只改兩份、同一次提交裡看得完，次數多而每一次都確認得了，不分層也行，代價是每週改兩次；十份清單、一年改一次，次數少而每次要確認十份、漏改會持續一年，到了需要一致性檢查的程度，改成分層的成本比加一道檢查低或相近。</p>
<p>主機由 Terraform 這類工具建立、而且每台主機都在 State 裡時，主機清冊的來源已經是 Terraform，inventory 由它產生；這時每台的清單也一起交給產生的步驟。repo 裡只有一份共用清單、每台只差主機名時，交給產生步驟的是每台的覆寫值，不是整份清單，上面「不需要分層」的判斷照樣成立。手寫的清單不會隨 Terraform 新增或刪除主機而增減，新開的主機沒有清單、已刪除的主機還留著一份，而清單與主機的對應只能靠人記得，所以 Terraform 每增刪一次主機，手寫的清單就有一次對不上的機會。</p>
<h2 id="延伸閱讀單台清單的格式與檢查腳本維護項目組態管理與-iac-的卡">延伸閱讀：單台清單的格式與檢查腳本、維護項目、組態管理與 IaC 的卡</h2>
<ul>
<li><a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>：本篇沿用的段名、<code>{{名稱}}</code> 的替換語法與 <code>--set</code>、檢查腳本的白名單、exit code、結果檔與 <code>systemd-run</code> 的限制</li>
<li><a href="/blog/devops/operations/09-host-lifecycle/long-running-host-maintenance/" data-link-title="長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算" data-link-desc="單台 Linux 主機（VPS、雲端主機）長期上線時，沒有人操作也會改變的狀態與維護對它們做的事：log 與資料的寫入者和清理機制（logrotate、journald、容器 log）、磁碟寫滿時各元件的反應、每份 log 要讀的欄位、入口在主機與雲端各層的盤點（ss）、各條更新管道的安全更新與重開機（dnf-automatic、unattended-upgrades、needs-restarting）、會到期與會改變的設定、檢查間隔的推算，以及接手一台已上線的主機時先處理哪幾項">長期上線主機的維護項目：會累積的資料、定期讀的 log、對外入口的盤點、安全更新、會到期的設定與檢查間隔的推算</a>：清單裡每一類檢查項目的來源，定期讀的 log 與對外入口的盤點方法；那一篇以單台主機為對象，依角色分組在本篇</li>
<li><a href="/blog/devops/infra/knowledge-cards/configuration-management/" data-link-title="Configuration Management（組態管理）" data-link-desc="要讓好幾台主機的套件、設定檔與服務維持同一個狀態、讀到 Ansible 的 inventory 與 group_vars、或要判斷主機上的設定該交給 Ansible 這類工具還是 Terraform 這類 IaC 管理時">Configuration Management（組態管理）</a>：inventory 的分層與合併規則、組態管理工具跟 IaC 的分界</li>
<li><a href="/blog/devops/infra/knowledge-cards/iac/" data-link-title="Infrastructure as Code (IaC)" data-link-desc="用程式碼描述基礎設施的最終狀態，由工具負責收斂現實與描述的差異">Infrastructure as Code (IaC)</a>：由 Terraform 建立主機、產生 inventory 的前提</li>
</ul>
]]></content:encoded></item><item><title>TOML</title><link>https://tarrragon.github.io/blog/devops/infra/knowledge-cards/toml/</link><pubDate>Mon, 26 Oct 2026 00:00:00 +0800</pubDate><guid>https://tarrragon.github.io/blog/devops/infra/knowledge-cards/toml/</guid><description>&lt;p>TOML（Tom&amp;rsquo;s Obvious, Minimal Language）是一種給人讀寫的設定檔格式：每一行是 &lt;code>鍵 = 值&lt;/code>，值的型別由寫法本身決定（字串要加引號、數字與布林值不加），用方括號的表頭把鍵分組，可以寫 &lt;code>#&lt;/code> 開頭的註解，沒有用縮排表示結構。規格由 toml.io 維護，1.0.0 在 2021 年 1 月發佈，1.1.0 在 2025 年 12 月發佈；Python 的 &lt;code>pyproject.toml&lt;/code>、Rust 的 &lt;code>Cargo.toml&lt;/code> 都用這個格式，跟 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/hcl/" data-link-title="HCL" data-link-desc="Terraform 使用的宣告式設定語言，用 resource block 描述基礎設施的目標狀態，工具負責收斂差異">HCL&lt;/a> 與 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/dotenv/" data-link-title=".env" data-link-desc="存放環境變數的純文字檔案，把機密值從程式碼分離出來">.env&lt;/a> 同樣是描述設定的檔案：比起 HCL，TOML 只描述資料、沒有變數與函式；比起 .env，TOML 有型別、分組與巢狀結構。&lt;/p>
&lt;h2 id="概念位置">概念位置&lt;/h2>
&lt;p>TOML 在設定檔格式裡的位置介於 JSON 與 YAML 之間：跟 JSON 一樣每個值的型別都由寫法明確決定，跟 YAML 一樣可以寫註解、適合人手動編輯。它只描述資料，需要變數、迴圈與條件來描述基礎設施時用的是 &lt;a href="https://tarrragon.github.io/blog/devops/infra/knowledge-cards/hcl/" data-link-title="HCL" data-link-desc="Terraform 使用的宣告式設定語言，用 resource block 描述基礎設施的目標狀態，工具負責收斂差異">HCL&lt;/a> 這類設定語言。它在本站被用來寫主機的維護清單，選它的理由與實測見 &lt;a href="https://tarrragon.github.io/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本&lt;/a>；Python 打包設定裡的 TOML 用法見 &lt;a href="https://tarrragon.github.io/blog/programming/python/advanced/07-packaging/pyproject-toml/" data-link-title="6.1 pyproject.toml 完整指南" data-link-desc="理解現代 Python 套件的設定標準">6.1 pyproject.toml 完整指南&lt;/a>。&lt;/p>
&lt;h2 id="可觀察訊號">可觀察訊號&lt;/h2>
&lt;p>需要讀懂 TOML 的情境包括：打開一個副檔名是 &lt;code>.toml&lt;/code> 的設定檔、要判斷某個值會被程式讀成字串還是數字、在 JSON、YAML、TOML 之間選設定檔格式，以及解析器回報 &lt;code>TOMLDecodeError&lt;/code> 這類錯誤、而同一份檔案在另一台機器上解析得過。&lt;/p>
&lt;h2 id="設計責任">設計責任&lt;/h2>
&lt;p>TOML 的基本結構：&lt;/p>





&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-toml" data-lang="toml">&lt;span class="line">&lt;span class="ln"> 1&lt;/span>&lt;span class="cl">&lt;span class="c"># 註解&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 2&lt;/span>&lt;span class="cl">&lt;span class="nx">name&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;web-demo&amp;#34;&lt;/span> &lt;span class="c"># 字串一定要加引號&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 3&lt;/span>&lt;span class="cl">&lt;span class="nx">port&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="mi">443&lt;/span> &lt;span class="c"># 整數&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 4&lt;/span>&lt;span class="cl">&lt;span class="nx">enabled&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="kc">true&lt;/span> &lt;span class="c"># 布林值只有 true 與 false&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 5&lt;/span>&lt;span class="cl">&lt;span class="nx">checked_at&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="ld">2026-10-26T06:21:00Z&lt;/span> &lt;span class="c"># 有時區的日期時間，解析成帶時區的時間物件&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 6&lt;/span>&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 7&lt;/span>&lt;span class="cl">&lt;span class="p">[&lt;/span>&lt;span class="nx">host&lt;/span>&lt;span class="p">]&lt;/span> &lt;span class="c"># 表格（table）：底下的鍵屬於 host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 8&lt;/span>&lt;span class="cl">&lt;span class="nx">provider&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;oracle-cloud&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln"> 9&lt;/span>&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">10&lt;/span>&lt;span class="cl">&lt;span class="p">[[&lt;/span>&lt;span class="nx">entrypoint&lt;/span>&lt;span class="p">]]&lt;/span> &lt;span class="c"># 表格陣列（array of tables）：每出現一次就新增一筆&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">11&lt;/span>&lt;span class="cl">&lt;span class="nx">port&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="mi">443&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">12&lt;/span>&lt;span class="cl">&lt;span class="nx">exposure&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;cdn-only&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">13&lt;/span>&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">14&lt;/span>&lt;span class="cl">&lt;span class="p">[[&lt;/span>&lt;span class="nx">entrypoint&lt;/span>&lt;span class="p">]]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">15&lt;/span>&lt;span class="cl">&lt;span class="nx">port&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="mi">22&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="ln">16&lt;/span>&lt;span class="cl">&lt;span class="nx">exposure&lt;/span> &lt;span class="p">=&lt;/span> &lt;span class="s2">&amp;#34;admin-only&amp;#34;&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;ul>
&lt;li>&lt;strong>&lt;code>[名稱]&lt;/code>&lt;/strong>：表格（table），指一組鍵值，等於程式裡的一個物件，跟試算表的格子無關；底下的鍵到下一個表頭為止都屬於它。&lt;/li>
&lt;li>&lt;strong>&lt;code>[[名稱]]&lt;/code>&lt;/strong>：表格陣列，每出現一次就在陣列裡加一筆，適合同一種項目要列很多筆的清單（入口、檢查項目、步驟）。&lt;/li>
&lt;li>&lt;strong>型別由寫法決定&lt;/strong>：&lt;code>&amp;quot;no&amp;quot;&lt;/code> 是字串，不加引號的 &lt;code>no&lt;/code> 不屬於任何型別，解析器報錯；YAML 1.1 的解析器（例如 PyYAML）把不加引號的 &lt;code>no&lt;/code> 讀成 &lt;code>false&lt;/code>（英文資料常稱為 Norway problem）、&lt;code>010&lt;/code> 讀成 8。寫法剛好屬於另一個型別時，TOML 一樣照那個型別讀：不加引號的 &lt;code>1.10&lt;/code> 是浮點數 1.1。所以版本號這類看起來像數字的字串要加引號。&lt;/li>
&lt;li>&lt;strong>日期與時間是原生型別&lt;/strong>：有時區的日期時間、沒有時區的日期時間、日期、時間各是一種型別，不必用字串存再自己解析。&lt;/li>
&lt;li>&lt;strong>巢狀的表格陣列&lt;/strong>：&lt;code>[[entrypoint.verify]]&lt;/code> 屬於它上面最近的那一筆 &lt;code>[[entrypoint]]&lt;/code>，讓每一筆項目各有自己的一組子項目。&lt;/li>
&lt;li>&lt;strong>寫在同一行的表格（inline table）&lt;/strong>：&lt;code>verify = { command = &amp;quot;...&amp;quot;, expect_stdout = &amp;quot;ok&amp;quot; }&lt;/code>，適合只有幾個欄位的子項目；TOML 1.0 要寫在一行內、結尾不能有逗號。&lt;/li>
&lt;li>&lt;strong>兩種字串&lt;/strong>：雙引號的基本字串處理反斜線跳脫（regex 的 &lt;code>\.&lt;/code> 寫成 &lt;code>\\.&lt;/code>），單引號的字面字串不處理任何跳脫（寫成 &lt;code>'\.'&lt;/code>）。&lt;/li>
&lt;li>&lt;strong>數字的分隔&lt;/strong>：整數可以用底線分隔位數，&lt;code>1_073_741_824&lt;/code> 等於 &lt;code>1073741824&lt;/code>。&lt;/li>
&lt;/ul>
&lt;p>跟其他格式的差別：&lt;/p></description><content:encoded><![CDATA[<p>TOML（Tom&rsquo;s Obvious, Minimal Language）是一種給人讀寫的設定檔格式：每一行是 <code>鍵 = 值</code>，值的型別由寫法本身決定（字串要加引號、數字與布林值不加），用方括號的表頭把鍵分組，可以寫 <code>#</code> 開頭的註解，沒有用縮排表示結構。規格由 toml.io 維護，1.0.0 在 2021 年 1 月發佈，1.1.0 在 2025 年 12 月發佈；Python 的 <code>pyproject.toml</code>、Rust 的 <code>Cargo.toml</code> 都用這個格式，跟 <a href="/blog/devops/infra/knowledge-cards/hcl/" data-link-title="HCL" data-link-desc="Terraform 使用的宣告式設定語言，用 resource block 描述基礎設施的目標狀態，工具負責收斂差異">HCL</a> 與 <a href="/blog/devops/infra/knowledge-cards/dotenv/" data-link-title=".env" data-link-desc="存放環境變數的純文字檔案，把機密值從程式碼分離出來">.env</a> 同樣是描述設定的檔案：比起 HCL，TOML 只描述資料、沒有變數與函式；比起 .env，TOML 有型別、分組與巢狀結構。</p>
<h2 id="概念位置">概念位置</h2>
<p>TOML 在設定檔格式裡的位置介於 JSON 與 YAML 之間：跟 JSON 一樣每個值的型別都由寫法明確決定，跟 YAML 一樣可以寫註解、適合人手動編輯。它只描述資料，需要變數、迴圈與條件來描述基礎設施時用的是 <a href="/blog/devops/infra/knowledge-cards/hcl/" data-link-title="HCL" data-link-desc="Terraform 使用的宣告式設定語言，用 resource block 描述基礎設施的目標狀態，工具負責收斂差異">HCL</a> 這類設定語言。它在本站被用來寫主機的維護清單，選它的理由與實測見 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>；Python 打包設定裡的 TOML 用法見 <a href="/blog/programming/python/advanced/07-packaging/pyproject-toml/" data-link-title="6.1 pyproject.toml 完整指南" data-link-desc="理解現代 Python 套件的設定標準">6.1 pyproject.toml 完整指南</a>。</p>
<h2 id="可觀察訊號">可觀察訊號</h2>
<p>需要讀懂 TOML 的情境包括：打開一個副檔名是 <code>.toml</code> 的設定檔、要判斷某個值會被程式讀成字串還是數字、在 JSON、YAML、TOML 之間選設定檔格式，以及解析器回報 <code>TOMLDecodeError</code> 這類錯誤、而同一份檔案在另一台機器上解析得過。</p>
<h2 id="設計責任">設計責任</h2>
<p>TOML 的基本結構：</p>





<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="ln"> 1</span><span class="cl"><span class="c"># 註解</span>
</span></span><span class="line"><span class="ln"> 2</span><span class="cl"><span class="nx">name</span> <span class="p">=</span> <span class="s2">&#34;web-demo&#34;</span>          <span class="c"># 字串一定要加引號</span>
</span></span><span class="line"><span class="ln"> 3</span><span class="cl"><span class="nx">port</span> <span class="p">=</span> <span class="mi">443</span>                 <span class="c"># 整數</span>
</span></span><span class="line"><span class="ln"> 4</span><span class="cl"><span class="nx">enabled</span> <span class="p">=</span> <span class="kc">true</span>             <span class="c"># 布林值只有 true 與 false</span>
</span></span><span class="line"><span class="ln"> 5</span><span class="cl"><span class="nx">checked_at</span> <span class="p">=</span> <span class="ld">2026-10-26T06:21:00Z</span>   <span class="c"># 有時區的日期時間，解析成帶時區的時間物件</span>
</span></span><span class="line"><span class="ln"> 6</span><span class="cl">
</span></span><span class="line"><span class="ln"> 7</span><span class="cl"><span class="p">[</span><span class="nx">host</span><span class="p">]</span>                     <span class="c"># 表格（table）：底下的鍵屬於 host</span>
</span></span><span class="line"><span class="ln"> 8</span><span class="cl"><span class="nx">provider</span> <span class="p">=</span> <span class="s2">&#34;oracle-cloud&#34;</span>
</span></span><span class="line"><span class="ln"> 9</span><span class="cl">
</span></span><span class="line"><span class="ln">10</span><span class="cl"><span class="p">[[</span><span class="nx">entrypoint</span><span class="p">]]</span>             <span class="c"># 表格陣列（array of tables）：每出現一次就新增一筆</span>
</span></span><span class="line"><span class="ln">11</span><span class="cl"><span class="nx">port</span> <span class="p">=</span> <span class="mi">443</span>
</span></span><span class="line"><span class="ln">12</span><span class="cl"><span class="nx">exposure</span> <span class="p">=</span> <span class="s2">&#34;cdn-only&#34;</span>
</span></span><span class="line"><span class="ln">13</span><span class="cl">
</span></span><span class="line"><span class="ln">14</span><span class="cl"><span class="p">[[</span><span class="nx">entrypoint</span><span class="p">]]</span>
</span></span><span class="line"><span class="ln">15</span><span class="cl"><span class="nx">port</span> <span class="p">=</span> <span class="mi">22</span>
</span></span><span class="line"><span class="ln">16</span><span class="cl"><span class="nx">exposure</span> <span class="p">=</span> <span class="s2">&#34;admin-only&#34;</span></span></span></code></pre></div><ul>
<li><strong><code>[名稱]</code></strong>：表格（table），指一組鍵值，等於程式裡的一個物件，跟試算表的格子無關；底下的鍵到下一個表頭為止都屬於它。</li>
<li><strong><code>[[名稱]]</code></strong>：表格陣列，每出現一次就在陣列裡加一筆，適合同一種項目要列很多筆的清單（入口、檢查項目、步驟）。</li>
<li><strong>型別由寫法決定</strong>：<code>&quot;no&quot;</code> 是字串，不加引號的 <code>no</code> 不屬於任何型別，解析器報錯；YAML 1.1 的解析器（例如 PyYAML）把不加引號的 <code>no</code> 讀成 <code>false</code>（英文資料常稱為 Norway problem）、<code>010</code> 讀成 8。寫法剛好屬於另一個型別時，TOML 一樣照那個型別讀：不加引號的 <code>1.10</code> 是浮點數 1.1。所以版本號這類看起來像數字的字串要加引號。</li>
<li><strong>日期與時間是原生型別</strong>：有時區的日期時間、沒有時區的日期時間、日期、時間各是一種型別，不必用字串存再自己解析。</li>
<li><strong>巢狀的表格陣列</strong>：<code>[[entrypoint.verify]]</code> 屬於它上面最近的那一筆 <code>[[entrypoint]]</code>，讓每一筆項目各有自己的一組子項目。</li>
<li><strong>寫在同一行的表格（inline table）</strong>：<code>verify = { command = &quot;...&quot;, expect_stdout = &quot;ok&quot; }</code>，適合只有幾個欄位的子項目；TOML 1.0 要寫在一行內、結尾不能有逗號。</li>
<li><strong>兩種字串</strong>：雙引號的基本字串處理反斜線跳脫（regex 的 <code>\.</code> 寫成 <code>\\.</code>），單引號的字面字串不處理任何跳脫（寫成 <code>'\.'</code>）。</li>
<li><strong>數字的分隔</strong>：整數可以用底線分隔位數，<code>1_073_741_824</code> 等於 <code>1073741824</code>。</li>
</ul>
<p>跟其他格式的差別：</p>
<table>
  <thead>
      <tr>
          <th>格式</th>
          <th>註解</th>
          <th>型別怎麼決定</th>
          <th>結構怎麼表示</th>
          <th>適合的場景</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>JSON</td>
          <td>不能</td>
          <td>寫法決定，規則嚴格</td>
          <td>大括號與中括號</td>
          <td>程式之間交換資料</td>
      </tr>
      <tr>
          <td>YAML</td>
          <td>能</td>
          <td>不加引號的值由解析器推測（版本之間規則不同）</td>
          <td>縮排</td>
          <td>巢狀很深的設定（Kubernetes、CI 的 workflow）</td>
      </tr>
      <tr>
          <td>TOML</td>
          <td>能</td>
          <td>寫法決定，不加引號又不是合法值時報錯</td>
          <td>表頭與表格陣列</td>
          <td>人手動維護、層數不深的設定與清單</td>
      </tr>
  </tbody>
</table>
<p><strong>版本對解析器的影響</strong>：1.1.0 新增的語法（寫在同一行的表格 inline table 可以跨行並在結尾加逗號、<code>\xHH</code> 與 <code>\e</code> 跳脫、時間可以省略秒）只有支援 1.1 的解析器讀得懂。Python 的標準函式庫 <code>tomllib</code> 在 3.11 加入時支援 1.0，3.15 起支援 1.1，3.10 以下沒有 <code>tomllib</code>、要另裝 <code>tomli</code>。同一份用了 1.1 語法的檔案，在 Python 3.15 解析得過，在 3.12 會報錯；版本對不上時的兩種結果與寫檔時的對策，見 <a href="/blog/devops/operations/09-host-lifecycle/host-manifest-in-toml/" data-link-title="主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本" data-link-desc="把一台主機的維護、暫停與恢復上線項目寫成一份資料檔：這份清單對格式的需求、JSON、YAML、TOML 逐項比對（YAML 把 no 讀成 false、010 讀成 8 的實測，以及 TOML 照樣把 1.10 讀成 1.1）、TOML 的表格與表格陣列寫法、1.0 與 1.1 的解析器差異（Python tomllib 的 TOMLDecodeError）、欄位怎麼依檢查腳本與人的需要設計，以及只讀檢查腳本的白名單條件、輸出與 exit code">主機維護清單寫成 TOML：JSON、YAML、TOML 的取捨、TOML 1.0 與 1.1 的版本差異、清單的欄位設計與只讀檢查腳本</a>〈TOML 1.0 與 1.1 的差異與解析器的版本〉。<code>tomllib</code> 只能讀不能寫，要由程式寫出 TOML 時需要第三方套件。</p>
<h2 id="鄰卡">鄰卡</h2>
<ul>
<li><a href="/blog/devops/infra/knowledge-cards/yaml/" data-link-title="YAML" data-link-desc="讀到 .yaml 或 .yml 設定檔、一個沒加引號的值被程式讀成布林值或別的數字、同一份檔案換一個解析器結果不同，或要在 JSON、YAML、TOML 之間選設定檔格式時">YAML</a></li>
<li><a href="/blog/devops/infra/knowledge-cards/hcl/" data-link-title="HCL" data-link-desc="Terraform 使用的宣告式設定語言，用 resource block 描述基礎設施的目標狀態，工具負責收斂差異">HCL</a></li>
<li><a href="/blog/devops/infra/knowledge-cards/dotenv/" data-link-title=".env" data-link-desc="存放環境變數的純文字檔案，把機密值從程式碼分離出來">.env</a></li>
<li><a href="/blog/devops/infra/knowledge-cards/php-ini/" data-link-title="php.ini / .user.ini" data-link-desc="PHP 的執行期設定檔，控制記憶體上限、上傳大小、錯誤報告等 runtime 行為">php.ini / .user.ini</a></li>
</ul>
]]></content:encoded></item></channel></rss>