Skip to main content

2 posts tagged with "Troubleshooting"

Following a real production incident all the way down to its root cause.

View All Tags

One Vietnamese Diacritic Killed an API Call: cf-ipcity, HttpClient and the ASCII Limit

· 10 min read
Nguyễn Huỳnh Minh Tiến
Middle Fullstack Developer @ Utop.vn
Summary

Cloudflare injects cf-ipcity into incoming requests, and for visitors in Vietnam the value is Hồ Chí Minh — with diacritics, which means non-ASCII. Our .NET service forwarded every incoming header verbatim onto its outgoing calls, so that value landed in HttpClient. The surprise is that Headers.Add does not throw, and TryAddWithoutValidation returns true; everything only blows up at SendAsync with HttpRequestException: Request headers must contain only ASCII characters, and not a single byte leaves the process. The bug is neither Cloudflare's nor .NET's — it is in an application that forwards every header unconditionally.

The setting is an e-commerce loyalty platform serving roughly three million customers. I have removed the client's name and every identifying detail; what remains is the technical part.

Everything looked normal. The API was running. The Kubernetes pods were healthy. The database was fine. Requests were reaching the application. But one HTTP call to an internal service kept failing in production — and only in production.

What broke it, it turned out, was the name of the user's own city.

ONNX Runtime Dies on Alpine: the ld-linux-x86-64.so.2 Error and Why libc6-compat Won't Save You

· 13 min read
Nguyễn Huỳnh Minh Tiến
Middle Fullstack Developer @ Utop.vn
Summary

onnxruntime-node ships prebuilt binaries built against glibc, and libonnxruntime.so.1 carries a DT_NEEDED entry pointing directly at ld-linux-x86-64.so.2 — glibc's own dynamic loader. Alpine uses musl, where that file does not exist, so dlopen fails at the library-loading step. Installing libc6-compat or gcompat does not fix it: they provide the loader file so you clear the first error, then die on the second one, __vsnprintf_chk: symbol not found. The practical answer is to change the base image to node:22-bookworm-slim, paying about 89 MB more for something that actually runs.

At 21:30 I added a Dockerfile for ClubDay — an event web app whose drawing game is scored by an ONNX model running on the server itself. The base image was reflex: node:22-alpine, because it is small.

The image built cleanly. The container came up. docker ps was green. Then, at the moment the server loaded the model, everything collapsed with an error line that never mentions ONNX:

Error loading shared library ld-linux-x86-64.so.2: No such file or directory
(needed by /app/node_modules/onnxruntime-node/bin/napi-v6/linux/x64/libonnxruntime.so.1)

At 21:44 I changed one line in the Dockerfile and everything worked. But the fourteen minutes in between are worth writing down, because that error line names a file you never installed, never declared in package.json, and which is simply always there on your development machine.