Backend high-availability​

API-based HA​

EdgeDB server accepts different types of backends by looking into the protocol of the --backend-dsn command-line parameter. EdgeDB supports the following DSN protocols currently:

• stolon+consul+http://

• stolon+consul+https://

When using these protocols, EdgeDB builds the actual DSN of the cluster’s leader node by calling the corresponding API using credentials in the --backend-dsn and subscribes to that API for failover events. Once failover is detected, EdgeDB drops all backend connections and routes all new backend connections to the new leader node.

Stolon is an open-source cloud native PostgreSQL manager for PostgreSQL high availability. Currently, EdgeDB supports using a Stolon cluster as the backend in a Consul-based setup, where EdgeDB acts as a Stolon proxy. This way, you only need to manage Stolon sentinels and keepers, plus a Consul deployment. To use a Stolon cluster, run EdgeDB server with a DSN, like so:

$ 
  

edgedb-server \
  --backend-dsn stolon+consul+http://localhost:8500/my-cluster

EdgeDB will connect to the Consul HTTP service at localhost:8500, and subscribe to the updates of the cluster named my-cluster.

Using a regular postgres:// DSN disables API-based HA.

Adaptive HA​

EdgeDB also supports DNS-based generic HA backends. This may be a cloud database with multi-AZ failover or some custom HA Postgres cluster that keeps a DNS name always resolved to the leader node. Adaptive HA can be enabled with a switch in addition to a regular backend DSN:

$ 
  
  

edgedb-server \
  --backend-dsn postgres://xxx.rds.amazonaws.com \
  --enable-backend-adaptive-ha

Once enabled, EdgeDB server will keep track of unusual backend events like unexpected disconnects or Postgres shutdown notifications. When a threshold is reached, EdgeDB considers the backend to be in the “failover” state. It then drops all current backend connections and try to re-establish new connections with the same backend DSN. Because EdgeDB doesn’t cache resolved DNS values, the new connections will be established with the new leader node.

Under the hood of adaptive HA, EdgeDB maintains a state machine to avoid endless switch-overs in an unstable network. State changes only happen when certain conditions are met.

Set of possible states:

• Healthy - all is good

• Unhealthy - a staging state before failover

• Failover - backend failover is in process

Rules of state switches:

Unhealthy -> Healthy

• Successfully connected to a non-hot-standby backend.

Unhealthy -> Failover

• More than 60% (configurable with environment variable EDGEDB_SERVER_BACKEND_ADAPTIVE_HA_DISCONNECT_PERCENT) of existing pgcons are “unexpectedly disconnected” (number of existing pgcons is captured at the moment we change to Unhealthy state, and maintained on “expected disconnects” too).

• (and) In Unhealthy state for more than 30 seconds (EDGEDB_SERVER_BACKEND_ADAPTIVE_HA_UNHEALTHY_MIN_TIME).

• (and) sys_pgcon is down.

• (or) Postgres shutdown/hot-standby notification received.

Healthy -> Unhealthy

• Any unexpected disconnect.

Healthy -> Failover

• Postgres shutdown/hot-standby notification received.

Failover -> Healthy

• Successfully connected to a non-hot-standby backend.

• (and) sys_pgcon is healthy.

(“pgcon” is a code name for backend connections, and “sys_pgcon” is a special backend connection which EdgeDB uses to talk to the “EdgeDB system database”.)