What is the Prometheus and Grafana ideal setup?

I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration?
The security and efficiency are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.

From your experience, what is the most common and useful configuration?
I see three main scenarios feel free to propose something else.

Thanks.

Solution I, Duplicate nothing

It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication.

```
+---------+ +------------+ +- Environment A -------------+
| Grafana | | Prometheus | | +-----------+ |
| |------>| |--+--+------>| Service 1 | |
+---------+ +------------+ | | | | | |
| | | +-----------+ |
| | | |
| | | +-----------+ |
| +------>| Service 2 | |
| | | | |
| | +-----------+ |
| +-----------------------------+
|
| +- Environment B -------------+
| | +-----------+ |
+--------->| Service 1 | |
| | | | |
| | +-----------+ |
| |

I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.

There're several factors which I think one has to consider:

• How cheap/expensinve a dedicated instance is?

• What are security concerns/considerations?

• What is expected volume of data from every environment?

This table can resume them :

|    Factor \ |  Solution I  | Solution II  | Solution III |
 |____________\|______________|______________|______________|
 | Data Volume |     Low      |     High     |     High     |
 |-------------+--------------+--------------+--------------|
 |    Security |      No      |     Yes      |     Yes      |
 |-------------+--------------+--------------+--------------|
 |Instance Cost|     High     |     Low      |    Medium    |
 |_____________|______________|______________|______________|

Based on the analysis of one's particular circumstances a preferred solution may be selected.

I would love to know what others think about the subject, though

StackExchange DevOps Q#10832, answer score: 3