host.docker.internal Is a Liar (On Linux)

I spent two days chasing a 5-second page load that had no business existing. Along the way I blamed the CPU governor, IPv6, disk contention, and a laundry list of side projects running on the same box. All of those theories were reasonable. None of them were it. The real answer was one bad assumption baked into a couple of connection strings, and it wore three different disguises before I finally cornered it. The gist of it: host.docker.internal is a liar (on Linux).

Here’s the whole mess, in order, mistakes included.

The setup

I’ve been building an app, and it was finally time to get it out of “runs fine on my dev machine” territory and into a real UAT environment. That meant moving it off a Windows desktop running Docker Desktop and onto a dedicated Ubuntu Server box running native Docker Engine. Same release image, just a handful of config overrides for UAT.

Along for the ride was the self-hosted observability stack I’ve also been assembling (GlitchTip, SigNoz, and OpenTelemetry, my attempt at something resembling Datadog or App Insights without the invoice). It wasn’t the point of the move, it was just riding shotgun with the app, which turned out to matter later, because it gave me a whole extra pile of red herrings to chase.

Page transitions that felt instant on my dev machine were taking about 5 seconds on UAT. That’s not a rounding error, that’s a “something is actually broken” gap. And the hardware didn’t explain it. My dev box runs an i9-13900K on NVME drives. The Ubuntu Server box is an older i7-10700 on an NVME drive. Sure, the dev machine is faster, but not “5 seconds slower” faster. Whatever this was, it wasn’t the processor.

The red herrings

Let’s walk through these first. Each one taught me something, even though none of them turned out to be the actual bug.

CPU governor

Ubuntu Server had defaulted to the powersave governor. That’s a real performance hit on a server that’s supposed to be doing work, so I fixed it:

cpupower frequency-set -g performance

Worth doing. Did not fix my problem.

IPv6 happy-eyeballs delay

I theorized for a while that the classic IPv6 happy-eyeballs delay was adding latency, and I talked myself into disabling it. Then I got distracted and never actually did it. This turned into its own small lesson: I kept debugging as if I’d made a change I’d only discussed making. Always double check what you actually did versus what you talked yourself into doing before you burn more time on a dead branch.

Disk and ClickHouse contention

The Ubuntu box was running a lot more than just this deploy was addressing. SigNoz, GlitchTip, Authentik, and a handful of unrelated side projects added up to around 20 containers. I figured ClickHouse or general disk contention made a reasonable culprit. docker stats disagreed. Peak CPU across the whole stack sat at 2%. I disabled SigNoz entirely just to rule it out completely, and the page load time didn’t budge. Infra theory, dead.

Back to basics

At this point I stopped guessing at the system level and went back to the request itself. I pulled up curl -w and started timing things directly, comparing a localhost hit against going through the actual domain.

That’s when I found the real smoking gun sitting in docker logs on the API container:

RabbitMQ.Client.Exceptions.BrokerUnreachableException
System.Net.Sockets.SocketException: Name or service not known

The connection string behind it: amqp://<user>:<pass>@host.docker.internal:5673/.

Root cause #1: host.docker.internal is a liar (on Linux)

Here’s the thing I didn’t know: host.docker.internal auto-resolves for free on Docker Desktop, whether you’re on Windows or Mac. On native Linux Docker Engine, it does not. You have to explicitly opt in per service:

extra_hosts:
  - "host.docker.internal:host-gateway"

This isn’t a bug. Linux runs Docker natively, there’s no VM sitting in between bridging things the way Docker Desktop’s VM does on Windows and Mac. But it’s exactly the kind of thing that only bites you the day you move off Docker Desktop, because it silently works everywhere until it doesn’t.

Detour: RabbitMQ wasn’t even running

While I was chasing the DNS error down, I noticed RabbitMQ itself was crash-looping. The logs were blunt about it:

Cookie file /var/lib/rabbitmq/.erlang.cookie must be accessible by owner only

This is a classic symptom of a data volume that ever touched a Windows filesystem. NTFS doesn’t preserve Unix permission bits, so the Erlang cookie file came out with permissions looser than the 600 that Erlang’s distribution protocol demands. The fix is a one-liner:

chmod 600 /var/lib/rabbitmq/.erlang.cookie

Or, if you’re on a single-node setup, just delete the file and let it regenerate.

Fixed everything. Still 5 seconds. Cue the “fml” moment

I fixed RabbitMQ. I fixed the DNS entry for the API. I reloaded the page, fully expecting to see that instant transition again.

Still 5 seconds.

So I went back to timing things, this time hitting the container directly to sidestep Caddy’s SSL redirect and get a cleaner read on where the time was actually going:

curl -w "%{time_starttransfer}s / %{time_total}s\n" http://localhost:49504

The numbers didn’t add up at all. Hitting localhost:49504 from the host took 11 seconds. Hitting the container’s real IP directly took 0.0007 seconds. Hitting the app from inside its own container took 0.0005 seconds. That first path had something seriously wrong with it, and none of my working theories explained an 11-second gap between “hit the host port” and “hit the container directly.”

The twist: I’d been testing the wrong port the entire time

Here’s the part that stung. I ran docker port to double check what was actually mapped, and it said 49505. Every single test I’d run up to that point had been against 49504.

A typo. A one-digit typo had sent me down a legitimate-looking but completely wrong path, IPv6 loopback theories and all.

Once I pointed my tests at the right container, web instead of api, I found the exact same failure pattern wearing a different costume: host.docker.internal again, this time in the connection string for Redis.

Same disease, different symptom.

Sweeping for the rest of the landmine

At that point I wasn’t going to wait around and discover every remaining instance one crash at a time. I swept the whole stack:

grep -rn "host.docker.internal" --include="*.yml" --include="*.yaml" --include="*.env*"

That turned up every other spot where I’d buried the same assumption, so I could fix all of it in one pass instead of playing whack-a-mole for another week.

The actual lesson

I’d baked one config assumption, “host.docker.internal just works,” into a handful of connection strings, and it worked fine in dev, so I never questioned it. The moment the environment changed, that assumption surfaced as three unrelated-looking failures: a crash-looping broker, a DNS resolution error, and a silent hang, plus one dumb typo that sent the whole investigation sideways for a while. Same root cause, three costumes.

Checking docker stats and chasing infra-level theories first wasn’t wrong. Governor, disk, IPv6, all reasonable things to rule out early. But the fastest actual path back to ground truth was simpler than any of that: check the logs of the specific container that’s slow, not the whole system.

Practical takeaways

• RTFL – read the fuggin logs. If I had started with logs instead of assuming it was an issue with the system, I would have saved myself a lot of headache and time.
• host.docker.internal auto-resolves on Docker Desktop, but not on native Linux Docker Engine. You need extra_hosts: ["host.docker.internal:host-gateway"] on every service that needs it.
• If a RabbitMQ data volume ever touched a Windows filesystem, check .erlang.cookie permissions before assuming a deeper problem. It needs to be 600, owner-only.
• Bisect the request path instead of guessing at layers. Run curl -w "%{time_starttransfer}s / %{time_total}s" from inside the container, then from the host, then through the proxy, and compare.
• The long-term fix isn’t routing back out through the host at all. Containers on the same Docker host should talk to each other over a shared Docker network using service names, like redis:6379, instead of leaning on host.docker.internal. That removes this whole bug class permanently.
  • The obvious caveat here — and the one that affects me — is that I want multiple other apps to reuse some of this infra rather than have n copies of Redis and RabbitMQ in my dev and UAT environments.
• Sanity check the exact port and hostname you’re testing against before you trust timing data that doesn’t match your mental model. A typo can eat a full day of debugging time and leave you convinced you’re looking at something exotic.

Additional Resources:

• Define services in Docker Compose | Docker Docs – docker.com (link specifically to extra_hosts)
• The Equivalent of –add-host=host.docker.internal:host-gateway in Docker Compose | Baeldung on Ops – Baeldung
• How to connect to the Docker host from inside a Docker container? | by Tim van Baarsen | Medium – Tim van Baarsen

Credits

Photo by Gabriel Heinzer on Unsplash