Handle OS and Software maintenance/updates on Hardware distributed to Customers

In the last weeks I have tried to find a solution to the following scenario assuming that someone else had similar requirements and documented his solution but all my searches were unsuccessful.
So now I'll just ask away and hope that someone with enough experience in this field can give me directions so that I don't take the wrong turn too many times.

The simplified scenario:

Let's say I have written a piece of software with a web interface and REST API for my customers which absolutely has to run in the local network at the customer's premises and will be provided to them as a "black box" hardware appliance. Apart from that I will essentially provide it "as a service" to them, meaning that maintenance and updates are my responsibility. I come to them with a small headless pre-installed and pre-configured PC, connect it to their network, perform only networking configuration and I'm done. The appliance is ready to communicate with any other software components running in the customer's network.

To be a bit more precise my software product runs under Linux (and also Windows), it should also be possible to build it as a Docker container (if that makes any sense for the proposed solution, although my know-how with Docker is still limited). I expect to set up CentOS 7 on the appliances.

Now I have two challenges:

• Keeping CentOS packages up-to-date

• ‎Keeping my software package up-to-date

To prevent all customers from experiencing the same bug at the same time I want to be in full control which customers update which packages at which point in time on the appliance (essentially a staged rollout).
In regards to my own software package this could very well be in form of different update channels (beta, stable, etc.).
Ideally I am also able to fetch and push configuration and other files from/to the appliance.
It would be great if I could use existing solutions for everything instead of rolling my own mechanisms.

Also, the solution should scale to hundreds or ev

I can provide some personal experience here. I worked at a company that had the same need. The product was a BeagleBone (like an Arduino or Raspberry Pi) that ran our software and sent data back to our SaaS. It had to be installed inside customer networks, and be a little black box that they didn't touch.

When I walked into this company, management of the systems was a nightmare. Networking was a nightmare. OS updates, keeping them online, shipping new versions of the software, etc... it was a nightmare.

The path we ended up choosing ended up working really well: The entire application and supporting packages was shipped as a Docker container. That allowed us to easily ship application code changes and control versions of any supporting software (ffmpg, gcc, etc...).

With the application abstracted, that simply left making the OS as minimal as possible. All we needed was a dead simple OS that could run docker. One of our low level engineers actually made our own in-house fork of Debian (if I remember) and we shipped that embedded directly into the devices.

We already had a service contract with all of our customers to service these machines yearly, so when our field technician went on-site, the procedure was to replace the entire unit with a fresh hardware that came with the latest embedded OS. We didn't have to make many OS changes just to run Docker, so often times swapping wasn't even necessary. But out of procedure, we always swapped hardware whether it was an emergency service call or the yearly maintenance.

I can't say if this would be a standard approach or not, but just that it worked really well for us and was a life saver. It sounds like your idea of shipping the OS as an immutable image is along the same lines.

Edit: When I left, we were still using home grown bash scripts to automate the Docker container deployments, but the idea was to move toward a config management tool (Like Chef, Salt, or Ansible). The basic premise is that you have a private Docker registry somewhere, then on the devices you would simply run a couple commands:

docker pull my-private-registry/my-custom-image:latest
docker run my-custom-image:latest

If you are going to always be using the 'latest' tag, you could then have your devices ship with a cronjob that has then always shut down their local running Docker container, and re-pull the 'latest' tag every week during a scheduled maintenance window.

StackExchange DevOps Q#3451, answer score: 2