In my article Running Google App Engine Behind Cloudflare, the goal is to get to a point where horizontal scaling can happen. One of the final barriers is the location of the images. WordPress stores the images in wp-content on the local machine.
Once this is separated the WordPress site can be somewhat easily horizontally scaled. There are a few methods that can be used to achieve this, particularly in Google Cloud.
- We could spin up more VMs and point them to the database
- Load the WordPress Docker image into a Kubernetes Cluster
- Run WordPress in Google App Engine
The actual method of horizontal scaling is out of scope for this document but this is the last barrier to get you to that decision.
I am going to give credit where it is due up front. Google’s tutorial on running WordPress on Google App Engine was a good starting point but not the first article I came across on this.
I came across this article from Kinsta which has some pretty good directions on a tool that looks extremely promising.
With anything WordPress, there is a plugin for it! Here are some options. We will choose one of these for this article.
- Google Cloud Storage Plugin – I have not seen much on this one.
- WP Offload Media – This one seems to have been around the longest but it will cost you to migrate existing content
- WP-Stateless – Seems extremely promising. This shows up 3rd on the list but is the one we will implement in this article.
For this article I decided it probably is not a good idea to make intrusive changes to this blog in order to generate more content. For this reason I decided to clone the production into staging.
Also, make sure to kick off a snapshot and backup of the database and VM beforehand.
We need to install and activate the WP-Stateless plugin.
Create Storage Bucket
WP-Stateless will guide you through creating the bucket but I wanted to do that manually to walk through the options. In the Storage / Browser section click “Create Bucket”. I had every intention of using a custom domain name but it does not appears to be supported according to Google.
With Cloudflare since I have it set to strict, it is expecting an origin cert. If I were to downgrade the site to flexible it would connect over 80 and likely work. This is a little bit concerning because Cloudflare’s CDN is great and this now bypasses it.
“Note: You can use ahttps://cloud.google.com/storage/docs/request-endpoints#cname
CNAMEredirect only with HTTP, not with HTTPS. To serve your content through a custom domain over SSL, you can set up a load balancer.”
This is my lab so I chose the cheapest option but since we cannot use behind a secure CloudFlare, you may opt for multi-region. Google Storage is not a full CDN but there are tools you can layer on top of it for that.
Next we will set up the ACLs. This is a mistake I made during the initial setup assuming there were no ACLs. The result was that the sync would claim to work but nothing would actually happen. I left this to show the config mistake.
It then runs you through a nice wizard. You can do a manual run as well. If you go that route, WP-Stateless’s instructions are fairly complete.
the first step asks you to login and it generates a json file for you so that it can authenticate.
In the configuration we need to set a few options. Namely stateless.
Due to the SSL issue we will leave domain blank. There is currently nowhere to upload the origin cert so a CNAME uses the A record’s SSL cert and would cause a cert mismatch. Being in strict mode, this won’t work but it may work in “full” or “flexible” since Cloudflare does not validate the cert in one case or use it in the other. If you are in “Full” or lower, give it a shot though!
Run a sync and you’re off!
Initially I was running on a micro instance with under 1GB RAM and it locked up and ran out of RAM. The default Bulk size is 1, you may need to go closer to 1. I re-ran this on a 1.7GB instance and ran with 1 and had no issues.
On the VM itself I validated images were removed.
$ find /var/www/html/wp-content/uploads/ | wc -l 441 $ find /var/www/html/wp-content/uploads/ | wc -l 42
There are still 42 images. We’ll track that down!
Some of these images did not have proper permissions. Since I manually synced the filesystems for this staging environment some of the newer images had incorrect permissions.
$ find /var/www/html/wp-content/uploads/ -ls 131221 4 drwxr-xr-x 3 www-data www-data 4096 Oct 25 19:22 /var/www/html/wp-content/uploads/ 131223 4 drwxr-xr-x 4 www-data www-data 4096 Nov 1 08:38 /var/www/html/wp-content/uploads/2019 131224 4 drwxr-xr-x 2 www-data www-data 4096 Nov 8 14:40 /var/www/html/wp-content/uploads/2019/10 131891 12 -rw-r--r-- 1 www-data www-data 9512 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/10/http_1_1-100x100.png 131849 4 -rw-r--r-- 1 www-data www-data 2835 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/10/logo-100x100.jpg 131866 12 -rw-r--r-- 1 www-data www-data 9200 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/10/K2167-100x100.png 131682 4 -rw-r--r-- 1 www-data www-data 3324 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/10/wh_header-100x100.jpg 131229 20 drwxr-xr-x 2 www-data www-data 20480 Nov 8 14:40 /var/www/html/wp-content/uploads/2019/11 147546 12 -rw-r--r-- 1 root root 8442 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/11/GAE-Default-768x117.jpg 147574 4 -rw-r--r-- 1 root root 3173 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/11/GAE-Cert-300x52.jpg 147550 8 -rw-r--r-- 1 root root 7206 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/11/CF-DNS-Only-1024x48.jpg 136683 8 -rw-r--r-- 1 root root 6938 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/11/SSL-CF-Origin-300x140.jpg 131918 4 -rw-r--r-- 1 root root 1604 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/11/CF-DNS-Only-150x49.jpg 147575 32 -rw-r--r-- 1 root root 31759 Nov 8 03:46 /var/www/html/wp-content/uploads/2019/11/mysql-purge-1024x186.jpg
Chown to the rescue
$ chown -R www-data:www-data /var/www/html/wp-content/uploads/*
Now we’re down to 8 after running again and these are actually unused images.
$ find /var/www/html/wp-content/uploads/ | wc -l 8
Here we have used a free plugin to move our images to a shared and central repository. Due to my configuration and desire to keep it, it does not leverage Cloudflare’s CDN but you are able to make your own decision on that.
UPDATE: 20191109 – I can confirm lowering Cloudflare security to Full and adding the CNAME to c.storage.googleapis.com does allow this to work. It would be a decision point at the time of needing this whether I go that route.
At this point, at least in this test environment, I could spin up multiple front ends now to handle any excess of traffic.
Another benefit of this is it helps keep your VM light without having to store all of your images on it.