Design and Evaluation of Self-Healing High-Availability PostgreSQL Clusters

The financial technology (FinTech) industry requires databases that provide strong consistency, high availability, and reliable data durability. Because of this, many organizations are moving towards multi-cloud database systems. However, traditional distributed consensus protocols such as Raft and Paxos experience major performance and latency problems when used across high-latency Wide Area Networks (WANs) in multi-cloud environments. In addition, existing PostgreSQL high-availability tools such as Patroni, a high-availability orchestrator that automates leader election, failover, and cluster management for PostgreSQL databases, are mainly designed to maximize availability instead of consistency. They also lack awareness of cloud boundaries, which can increase the risk of data loss and split-brain situations during failures. To address these problems, this work introduces a new architecture that separates a global multi-cloud cluster into smaller independent clusters, each running within a single cloud provider. A new Global Coordination Layer (GCL), developed in Kotlin and based on Apache Ratis, a Raft-based consensus library for distributed systems, is used to manage global consensus, coordinate failovers, and prevent split-brain scenarios. At the same time, Patroni was modified to enforce strict cross-cloud synchronous replication, changing its behavior from prioritizing availability to prioritizing consistency. The system was tested under different fault conditions, including simulated network delays of up to 250 milliseconds and complete cloud failures. The results showed that the GCL-based architecture reduced the total recovery time before the database became writable again when compared to a traditional stretched etcd-cluster design. The architecture also proved to be highly reliable during chaos engineering tests, producing zero false failovers and recovering non-synchronous replicas more effectively than the baseline solution. Overall, this work demonstrates that financial institutions can achieve strong consistency, high availability, and multi-cloud resilience using a traditional PostgreSQL master-slave architecture combined with local consensus systems and a Global Coordination Layer. This approach avoids the additional complexity commonly associated with multi-master sharded databases.