gamer-net.com · Post-mortem · Dune: Awakening self-hosting

My self-hosted world vanished from the server browser. Everything said it was healthy.

The battlegroup reported Healthy. The gateway registered with Funcom. The director heartbeat came back Request successful every thirty seconds. The world was still invisible in the Experimental tab — because update had been lying about being up to date for eleven days.

By strider916 · Alpine guest on Proxmox · k3s · app 4754530 · written up after roughly a week of dead ends

triage — start here ~$ why-is-my-world-gone
Is the battlegroup PHASE Healthy?
├─ no → pods first. Check image tags / webhook certs
└─ yes
   Is the director heartbeat succeeding?
   ├─ no → token or FLS auth
   └─ yes
      Do OTHER self-hosted worlds show in the list?
      ├─ no → Funcom-side outage. wait.
      └─ yes, but not mine
         → you are almost certainly on a stale build.
         → check StateFlags in appmanifest_4754530.acf

The headline bugsteamcmd gets wedged, and update reports success anyway

This is the one worth knowing about, because it fails silently and the symptom appears somewhere completely unrelated: your world quietly stops appearing in the in-game server browser.

Steam tracks install state for the self-hosted server package (app 4754530) in an appmanifest file. If an update is ever interrupted, that file can get stuck with StateFlags "6" — an "update stopped" state. In that condition steamcmd refuses to download anything, but the wrapper script interprets the result as a clean run and prints the reassuring lie:

Success! App '4754530' already up to date.

Meanwhile the manifest itself was explicit about the problem the entire time:

"StateFlags"      "6"          ← stuck / interrupted
"buildid"         "24028730"   ← what you have
"TargetBuildID"   "24204075"   ← what you should have
"BytesDownloaded" "0"

So the server kept running an eleven-day-old build while the game client patched forward. Funcom stops listing worlds whose build has drifted — which is why every other self-hosted world in the Experimental tab was visible and mine was not.

The trap: the failure mode is a false negative on a check you'd naturally trust. Once wedged, update will keep telling you you're current, forever, while you fall further behind with every client patch.

Diagnosis — run this before anything else

sudo grep -E "StateFlags|buildid|TargetBuildID" \
  /home/dune/.dune/download/steamapps/appmanifest_4754530.acf
StateFlags "4"
healthy
buildid == TargetBuildID
genuinely up to date
StateFlags anything else
wedged — fix below
buildid != TargetBuildID
you are behind, regardless of what update says

The fix

# 1. stop the world
sudo /home/dune/.dune/download/scripts/battlegroup.sh stop

# 2. back up while stopped (backups are I/O heavy)
sudo /home/dune/.dune/download/scripts/battlegroup.sh backup

# 3. move the wedged manifest aside — do not delete, you may want it back
sudo mv /home/dune/.dune/download/steamapps/appmanifest_4754530.acf \
        /tmp/appmanifest_4754530.acf.bak

# 4. update. steamcmd re-verifies (~5GB) then actually downloads
sudo /home/dune/.dune/download/scripts/battlegroup.sh update

# 5. start
sudo /home/dune/.dune/download/scripts/battlegroup.sh start

With the manifest gone, steamcmd treats the app as unrecognised, verifies what's on disk, and pulls the delta properly. The run that had been instantly failing now did the real work:

Update state (0x5) verifying install, progress: 93.78
Update state (0x61) downloading, progress: 99.98 (5202242632 / 5203291208)
Success! App '4754530' fully installed.
...
Downloaded version 2036754-0-shipping      ← was 2025705-0-shipping
Battlegroup: sh-xxxx updated to 2036754-0-shipping

Rollback, if step 4 makes things worse: sudo mv /tmp/appmanifest_4754530.acf.bak /home/dune/.dune/download/steamapps/appmanifest_4754530.acf

Verify the new build actually landed

NS=funcom-seabass-<your-battlegroup>
BG=<your-battlegroup>

# real image fields — expect the new version
sudo k3s kubectl get battlegroups.igw.funcom.com -n $NS $BG -o yaml \
  | grep "image:.*seabass-server:"

# images present in containerd — expect 6
sudo k3s ctr images ls | grep -c "2036754"

Counting gotcha: a plain grep -c across the whole CR will return equal counts for old and new versions. That's the kubectl.kubernetes.io/last-applied-configuration annotation holding a stale JSON copy of the entire spec. It's cosmetic — the operator reads the real fields. Grep for image: lines specifically.

Everything else that went wrongFour more undocumented failure modes

Getting to the manifest bug meant clearing a series of other problems first. Each one is worth knowing on its own — especially if you're running on a hypervisor Funcom doesn't support.

Symptom — Get-VM : The term 'Get-VM' is not recognized

Cause. The Windows-side battlegroup.bat is hard-coded for Hyper-V. battlegroup.ps1 calls Get-VM looking for a VM literally named dune-awakening. On Proxmox, KVM, ESXi or anything else, that cmdlet doesn't exist and the tool is useless to you.

Fix. Don't press a (initial-setup) — it will try to build a fresh Hyper-V VM on your Windows box. Manage the battlegroup from inside the Alpine guest instead. The real tooling ships in the Steam payload and lands at ~/.dune/download/scripts/: battlegroup.sh, setup.sh, and the bg-util binary. Everything the .bat does, these do.

Symptom — error converting YAML to JSON: yaml: control characters are not allowed

Cause. The generated battlegroup CR picks up stray control characters — typically CRLF line endings, often from a pasted token. Funcom's shipped files are known to carry CRLF in places.

Why it cascades. The CR apply fails, so the BattleGroup resource is never created, so every later step fails with confusing follow-on errors: battlegroups.igw.funcom.com "sh-xxxx" not found, then must specify --patch or --patch-file, then No filebrowser pod found. Four errors, one cause.

# find the offending bytes without dumping your token
grep -n -P '[\x00-\x08\x0B\x0C\x0E-\x1F]' ~/.dune/<world>.yaml

# strip carriage returns
sed -i 's/\r$//' ~/.dune/<world>.yaml
Symptom — pods in ImagePullBackOff, lookup registry.funcom.com: no such host

Cause. Images are pre-loaded into containerd locally; nothing is meant to be pulled from Funcom's registry (it isn't publicly resolvable). If a pod asks for a tag that wasn't loaded, k3s falls back to the network and fails. The giveaway is the placeholder tag 0-0-shipping where every loaded image is <version>-0-shipping.

This happens when the version-stamping step fails — which, in my case, was itself downstream of the control-character error above.

Fix. Rewrite the tags on the live resource. Reading and replacing in one pipeline avoids stale-file and resourceVersion problems:

sudo k3s kubectl get battlegroups.igw.funcom.com -n $NS $BG -o yaml \
  | sed 's/0-0-shipping/2036754-0-shipping/g' \
  | sudo k3s kubectl replace -f -

Don't try this by exporting to a file and applying it — the export's embedded annotation is an older, incomplete snapshot, and apply will fail with resource name may not be empty.

Symptom — failed calling webhook "mbattlegroup.kb.io": x509: certificate signed by unknown authority

Cause. The battlegroup operator runs a mutating admission webhook. cert-manager issues its serving cert and the CA injector patches the matching bundle into the webhook config. After restarts or partial runs those two drift apart, and the API server rejects the connection.

Fix. Restart the operators so the cert and CA bundle regenerate together, then retry:

sudo k3s kubectl rollout restart deployment -n funcom-operators
# wait ~60s for all four to return 2/2 Running

Note this error appears as error when creating — meaning the resource was rejected, not created. If a duplicate attempt left an empty namespace behind, delete it; the CR itself won't exist.

Performance1000ms ping on a LAN server, and the slow creep after it

The loopback problem

Freshly built worlds ship with HOST_DATACENTER_IP_ADDRESS set to 127.0.0.1. That value is handed to clients as part of the session path, so even a client sitting on the same switch gets routed the long way around. Raw ping to the VM was under 1ms; in-game latency was ~1000ms.

It appears in three places (director, server gateway, text router) and the change-battlegroup-ip action didn't commit it in my case. Set them directly:

sudo k3s kubectl get battlegroups.igw.funcom.com -n $NS $BG -o yaml \
  | sed 's/value: 127.0.0.1/value: 10.10.0.69/g' \
  | sudo k3s kubectl replace -f -

# verify — all three should read your LAN IP
sudo k3s kubectl get battlegroups.igw.funcom.com -n $NS $BG -o yaml \
  | grep -A1 "HOST_DATACENTER_IP_ADDRESS" | grep "value:"

Pods bake this in at startup, so restart afterwards. Confirm it took by reading the gateway's registration call — HostDatacenterIpAddress and GameRmqAddress should both show your LAN IP.

Latency that returns after about a day

Distinct problem, different signature: fixed at first, then creeping back to 300–400ms after ~24 hours of uptime. CPU sat at 96% idle throughout, so it wasn't compute.

The Hagga Basin instance runs against a 12GB cgroup limit. With swap at zero, a long-running server works right at that ceiling — constant allocation and reclaim with no cushion — and the simulation tick slows. Clients read that as latency.

  • Immediate: restart the battlegroup. Clears it every time.
  • Durable: battlegroup.sh enable-experimental-swap — officially documented as significantly reducing map memory requirements by allowing disk swap.
  • Better: more RAM. 31GB is under the comfortable tier; the VM must be fully powered off to resize, a guest reboot won't do it.

ReferenceThings that cost me hours to establish

Rotating the self-hosting token

There is no change-token action. The JWT is embedded in ~34 places in the CR and in the server-gateway-secret. Miss either and you get inconsistent auth. Keep both tokens out of shell history by using variables and a file:

# capture old from the live CR; new from a file you paste into
OLD=$(sudo k3s kubectl get battlegroups.igw.funcom.com -n $NS $BG -o yaml \
  | grep -o 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9\.[A-Za-z0-9._-]*' | head -1)
NEW=$(cat /tmp/newtoken)

# sanity: non-zero, similar length, actually different
echo "old:${#OLD} new:${#NEW}"

# target 1: the CR
sudo k3s kubectl get battlegroups.igw.funcom.com -n $NS $BG -o yaml \
  | sed "s|$OLD|$NEW|g" | sudo k3s kubectl replace -f -

# target 2: the secret (stringData lets k8s do the base64)
sudo k3s kubectl patch secret server-gateway-secret -n $NS --type=merge \
  -p "{\"stringData\":{\"FuncomLiveServices__ServiceAuthToken\":\"$NEW\"}}"

unset OLD NEW; shred -u /tmp/newtoken

Check the new token's HostId matches your world before swapping — it's in the JWT payload, and your namespace is derived from it. Different HostId means a new world, not a patched one.

cut -d. -f2 /tmp/newtoken | tr '_-' '/+' | base64 -d; echo

Reading status without chasing ghosts

DATABASE: Modifying
still initialising — don't start yet
db-dbdepl-util: Completed
correct. it's a one-shot job
Stopped, but DB pods running
correct. stop suspends the world, not the database
Terminating → Error → Running
normal restart churn, not a crash
ln: ... File exists
cosmetic symlink warning, ignore
GameRmqSecret in logs
rotates on every restart, don't panic

What renaming does and doesn't do

spec.title is the DisplayName in the browser and nothing else. Characters key off the BattlegroupId and live in the Postgres PV, which survives restarts, updates and renames. Dots in the name are fine — plenty of listed worlds have discord.gg in theirs.

PostscriptWhat I'd tell myself at the start

Two things would have saved most of a week.

First: when your world stops listing, check the Steam manifest before anything else. Not the token, not the display name, not the datacenter IP. The manifest. It's the only failure here that actively misreports itself, which makes it the one you'll rule out last if you go in order of plausibility.

Second: a healthy heartbeat is not the same as being listed. I spent a long time treating Request successful as proof the server side was blameless. It only proves Funcom accepts your token and considers the world alive. Whether it appears in the browser depends on other things — build version among them.

The rest of it — the CRLF characters, the placeholder image tags, the webhook cert drift, the loopback datacenter address — those are all first-boot problems you clear once. The manifest bug is the one that will come back, quietly, every time an interrupted update leaves it wedged.