Building a Magento Cluster

Building a Magento Cluster

I wanted to upgrade our Magento 1 web server from a simple single beefy a server running Centos6, Apache, Mysql, PHP-FPM and Redis on the same instance to a cluster of servers, with the objective to improve performance, add redundancy and enable easy scaling.

On one hand I could just grab a Cloudways instance and when I need more cpu's just move a magic slider. However I wanted more control and to see what kind of performance I could get splitting out the services.

This setup will be copied for  M2 but probably with some changes to caching.

Some Research

Scaling Magento Presentation - PUBLIC
Scaling Magento Reid Parham, Aaron Edmonds, and Kyle Terry How would you build the world’s largest, fastest, most complex Magento ecommerce store? Join three COPIOUS engineers as they share their approaches to this problem. This one-hour presentation will include the best practices, code samples,...

The best practice Adobe setup is overkill for what I want.

Reference architecture | Adobe Commerce Developer Guide
Adobe Commerce Developer Guide

Although old this Stack Exchange is a good read along with the GitHub docs on building out a big cluster

Magento configuration in a clustered environment
I’m currently working on a large Magento (Enterprise Edition) project where the client wants to use four load balanced web servers and four MySQL servers (one master, three slaves). I have not bee...
A highly available, fault tolerant, distributed, and load balanced LEMP cluster.
A highly available, fault tolerant, distributed, and load balanced LEMP cluster. - 0: Magento 1.12 Enterprise multi-store cluster

Most of the articles out there are older and geared towards data centres and less cloudy, whereas I want to do this fully in Google Cloud Platform. I have moved away from AWS due to better pricing on GCP, and I also find GCP easier and nicer to use.

For phase one I am not going to bother with a load server to split into multiple app servers, but instead split the admin out onto a separate server. So there will be two Compute Engine instances running Debian & NGINX for Magento and the admin panel, a Cloud SQL instance to run the database, a Memorystore instance to run the Redis cache for sessions, and a Filestore instance for the media folder.

On paper I think this looks like a really powerful setup and much simpler than the recommended from Adobe.

  • I don't need a slave database as I can have Cloud SQL with high availability in multiple zones with a single checkbox.
  • I am not doing to bother with another cache in front yet but try and use Cloudflare caching.

It's also nice to have a bucket that I use for moving sql dumps around.

Simple Architecture

Phase two I can add more Magento servers and add a cache and load balancer in front of everything.

Getting Started

I signed up for a new .dev domain with Google Domains, attached the domain to Cloudflare and created a project in Google Cloud Platform.

I set the name servers in Google Domains to Cloudflare

Because it's a dev domain HSTS must be turned on. I went through Cloudflare turning on strict SSL, redirecting to HTTPS and DNSSEC and also added SPF records.

Google Cloud Platform

I then created a new project in GCP and started adding the services I needed.


I then created a new private bucket to move the database dump and media folder from the old server to GCP

Compute Engine

I created a new instance with decent specs on the EPYC platform.

Cloud SQL

And a new database with high availability.

And set it to private IP so it is only accessible inside the Google VPC.

Then I could create and empty database and import the sql dump I had put into the bucket.

Installing NGINX & PHP

While the database is importing I can go back to the VM and setup NGINX, PHP and pull over the Magento repo and media folder.

I won't bother setting up SSH keys in my WSL Debian install and just use the web SSH for the moment.

sudo apt update && sudo apt upgrade

sudo apt install nginx php-fpm