Y’all remember how I posted some weeks ago about the issues we were having with the semi-regular crashing of the Apache webserver on our home network?

Well, I finally tracked down what seems to be the root cause. But it took a lot of steps to get there.

My usual strategy for solving computer-related problems, googling the hell out of variations of phrases about said problems, was failing me. I ultimately wound up having to talk to folks on the Debian support channel on IRC, of all things. I’d never installed or used an IRC client before, and certainly haven’t felt the need to ever track one down. Particularly in this current age of Discord and Slack and other rather fancier ways of getting your chat on.

But I gotta say, a few conversations with the folks on the Debian channel did finally point me in the right direction.

There were two big overall issues I had to solve to get constructive information about the crashing, and those issues were:

1. Make the system actually capable of recording coredumps when the crashes occurred
2. Install appropriate debugging symbols packages to be able to get useful information out of the coredumps

Item one on this list basically meant, set the right things so that when Apache horfed up a hairball, a suitable record of this was put somewhere I could find it. Digging through the logs in /var/log/apache2 hadn’t gotten me anywhere, so I needed that actual coredump, the big spew of information that would tell me exactly what fell over and why.

Since we’re running Debian 10.2 on our system, this meant telling Debian and Apache that I needed coredumps if Apache fell over, and where to actually put them. As I wrote in my earlier post, I’d apparently done some of the work to make this happen. But not all of the work.

What ultimately got Apache creating coredumps seemed to be a combo of making systemd able to dump core, and making Apache know where it had to put that dump. Which led me to item two: how to actually make that information useful.

This required me to install two debugging symbols packages: one for Apache, and one for PHP. These were necessary to make the information in the coredump human-readable and useful.

And that, finally, led Dara and me to discover the probable culprit: something called opcache.

Opcache, for those of you who aren’t technically inclined, is a thing intended to speed up performance of webpages written in PHP. It does this by taking the PHP code written by programmers and compiling it into computer-level code that can be executed a lot faster–and then caching that compiled code so that it can be loaded more quickly. End result for the user, a page loads faster because the server doesn’t have to work as hard to show it to you.

Which is delightful and all. When it actually works.

On our system, for whatever reason, it’s making things crash. We don’t know why yet. And until we know why, we’ve turned off opcache for the time being. It’s not a performance hit for us as far as we can tell. We didn’t have it before when we were running the older version of PHP anyway, so we’re not losing performance by having it off.

Eventually, we’ll figure out whether we can actually effectively use it. For the time being, though, we’d rather have a server that doesn’t crash on us every day!

This doesn’t mean I’ll be shifting my author headquarters back to our server, though. I’m going to keep it here on angelahighland.info, because even when our webserver is stable, we do still have periodic power problems. Particularly during windstorm season. But I’ll post further computer-geekery type things on annathepiper.org, and if you’re interested in those posts you can find recent ones on the sidebar on most pages of this site.

Let’s hear it for server stability!