Two server failover in nomad

I am just learning about orchestrators and it's my first time using any orchestrator (Nomad in this case).

Our software (video stream processing related) is installed on user premises, currently we have been using docker compose to deploy our software (bunch of services as docker container).

Now we have a client who needs failover and has provided us two physical server, one is to be used as a main server and other as failover. What we'd like to do is have our services installed on both the servers but only run these services on failover when the main server goes down.

We won't like to make any changes to our code. we have one services which process the video streams and raise events (VA service) and another service which handles database, users, events etc. (Client Service). The client service knows about this VA service and it assigns work to it. We don't want these services to be running on failover server when the main server is up since it will cause streams to be processed twice i.e produce duplicate events.

We are configuring our database in master-master mode so that data is available on failover when main fails ( trying it on postgres (current database), had previously done it on Mysql ).

I have set up nomad on both machines with nomad server and client both on each machine and tried configuring them such that if the machine with nomad leader server and its client shuts down, nomad server on failover should take charge and run tasks on its client, but i am unable to achieve this. If i shutdown the main server nothing happens on failover machine.

```
# Leader config file
datacenter = "dc1"
data_dir = "/opt/nomad/data"
bind_addr = "192.168.1.38"
name = "server1"

server {
enabled = true
bootstrap_expect = 2
server_join {
retry_join = [
"192.168.1.38",
"192.168.1.20",
]
retry_max = 0
retry_interval = "15s"
}
}

client {
enabled = true
servers = [
"192.168.1.38",
"192.168.1.20",

I think you are conflating a few concepts here, let me try to separate them in your mind.

First, separate availability of the platform, from the availability of your applications:

Nomad, like many other distributed systems, uses the raft consensus algorithm to determine who is the leader of the server cluster. This is different to older high-availability strategies which provide multiple endpoints served by a load balancer for example.

If you have two servers, it's impossible to elect a leader since none of the servers has a winning vote. For this reason, there is zero fault tolerance with two servers -- see https://developer.hashicorp.com/nomad/docs/concepts/consensus#deployment-table. If the active server goes down, requests cannot be processed.

So, if you want fault tolerance, put another Nomad server in the cluster, and bring it to 3. Else having 1 or 2 servers won't increase fault tolerance. The state is replicated across servers, so you be able to perform read actions, but you won't be able to change the state until a quorum is re-established and a new leader is elected.

Now, for you application:

You have Nomad clients defined on two machines, which means that you can submit jobs with services and tasks to those resources.

What we'd like to do is have our services installed on both the servers but only run these services on failover when the main server goes down.

What you are looking for here is the migrate stanza of the job - when Nomad determines that a node on which a job is running has failed, it (Nomad server) will migrate the job to a node which is up. However, as discussed before, if the server is unavailable because it has lost consensus, this won't be possible... So either you have an extremely reliable single machine running Nomad server, or you have a fault-tolerant 3-node cluster running Nomad server, with unreliable clients.

In terms of your VAService and client jobs -- you should write a Nomad job description which defines these tasks. They can expose services and Nomad's service discovery will connect them, so that they can run on different machines, etc.

We don't want these services to be running on failover server when the main server is up since it will cause streams to be processed twice i.e produce duplicate events.

Nomad is the orchestrator, therefore it determines where to place the job. Unless you have a count in the group which the task is running in greater than 1 (ie, more than one instance of the application placed in a group), then Nomad will schedule only one instance of it and you won't have to worry about double-processing streams. In terms of availability , you should choose a different strategy than the traditional HA passive-active :

• reschedule -- reschedule the group or job if a task fails or dies

• restart -- similar to reschedule, but just restarting in place -- see also check_restart

• migrate -- move the job to a different node if the client is marked for draining

These are definitions of how Nomad should treat failures in the operational lifecycle of the application.

Take a scenario where you have one server and clients enabled on it and the other machine. You write a Nomad job description with:

• 2 groups -- one for the server, one for the client

• each group has a reschedule, restart and check_restart stanza to deal with failures.

You deploy this job description to Nomad. The application server group gets placed on the same node where the Nomad server is running and the application client gets placed on the other machine. Everything works fine.

Now:

Failure scenario 1: the machine running only the client dies

• Nomad will reschedule see that the node is down and reschedule the group on the remaining machine.

• you have resilience, but there is a disruption in availability.

Failure scenario 2: the machine running the Nomad server dies

• Nomad cannot reschedule the job because quorum has been lost

• Application server is lost, but application client keeps running

Clearly neither of these is a good situation, so:

I would go with a resilience scenario such as: Use Nomad servers in a different failure domain to the Nomad clients.

If you are able to add a third node to the cluster, you will have fault tolerance at the cluster level. By using the built-in mechanisms of Nomad for rescheduling, restarting, etc job as discussed above, you will have a resilient application. You cannot eliminate downtime however without changing the application itself, either by making the server idempotent (ie, multiple processing requests against the same event do not change the event result), or by including some mechanism for the server itself to elect a leader of that tier of the application. There are several approaches here, but I think they'd fall outside of the scope of the question.

Finally, I would suggest opening this topic at the Hashicorp discussion forum so that you can get some deeper context and better feedback :)

StackExchange DevOps Q#17750, answer score: 2