Troubleshooting A Broken Website after upgrading from Ubuntu 18.04 to 22.04

Channeling administrators everywhere, last Friday I decided it would be best if I upgraded one of my servers from 18.04 to 22.04 on a Friday at 6 pm. How hard can it be?

Always turns out harder than I expect.. hours later, here is how it worked out..

The Troubleshooting Process

In my instance, it came down to a configuration that I had completely forgotten about.

Because this was a web server, we want to start by verifying if everything was running as it should be. The web server, and database, were fully operational so that wasn’t the issue. Understanding that the server and web server applications were operational we turn our attention to the web application.

Being a log guy, the first place we stop is the sites error.log, where we’re greeted with this error:

At first, I was confused. Umm, why is every request routing it through localhost (127.0.0.1) and via this 9003 port?

I turned my attention to my trusty advisor, Google, but wasn’t having much luck. Then I turned my attention back to the error log and focused on the program causing the issue, annotated here [proxy:error] and [proxy_fcgi:error]. This got me thinking about a time I was playing with FastCGI Process Manager (FPM) to help pool users and PHP for specific application.

Shoot, did I do this on this server? Opening the config for the site verified I was:

Ok, so now I knew what was happening. For whatever reason, after upgrading to Ubuntu 22.04 FPM was no longer working. Shoot, how’d I configure that before?

After some back and forth, focusing on what FPM was doing helped resolved the issue. When upgrading to 22.04 PHP is upgraded to 8.1 which means you have to update your environment to account for this. This means updating Apache to run with PHP 8.1 and updating the configurations to account for this change.

Here is an update you’ll have to run for PHP 8.1 on your new Ubuntu 22.04 instance:

apt install php8.1-fpm php8.1 libapache2-mod-php8.1 php8.1-common php8.1-mysql php8.1-xml php8.1-xmlrpc php8.1-curl php8.1-gd php8.1-imagick php8.1-cli php8.1-imap php8.1-mbstring php8.1-opcache php8.1-soap php8.1-zip php8.1-intl php8.1-bcmath unzip -y

We also need to verify the appropriate modules are enabled:

a2enmod mpm_event proxy_fcgi setenvif

Verify 8.1 FPM is running:

Then we need to move our old FPM configuration from the old PHP instance to the new one. Something as simple as:

cp /etc/php/7.4/fpm/pool.d/[domain].conf /etc/php/8.1/fpm/pool.d/[domain].conf

Then enable the 8.1 FPM config:

a2enconf php8.1-fpm

Verify there is nothing wrong with the config with Apache:

apachectl configtest

For good measure, remove any mods leveraging old versions of PHP

Then restart the web server:

service apache2 restart

If all goes as planed, this should be it. For me, the issue went a step further as one of the plugins I was running was not adequately supporting 8.1 but the end results was this site was operational again:

Upgrading is Always a Dangerous, and Scary, Activity

Whether it is at the application, or server, level I think that we sometimes fail to appreciate how chaotic, and stressful, upgrading can be to a user online. Even for someone that is technical, it can take a minute (or a few hours) to figure out what went wrong.

While this article took a few minutes to write, it created a stressful 4 hours Friday night as I tried to remember and figure out what was going on. I’m also a bit more technical than most and can usually find my way out of a bad spot, and if it gets hairy I can dial a friend, but this is not the case for many.

All this to say, let’s be more thoughtful when we’re working with online users and maybe be a bit more understanding when they share reservations about upgrading. It’s never as easy as we would hope. :)

Hope the above article helps if you’re encountering something similar.