monitoring/alerting system which can support very arbitrary data source and logic

I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.

(following is a made up example to more easily demonstrate my needs)

I have a database where I can find out the query execution times for my clients(which are individual companies). The query times are expected to vary greatly between clients, because of the data of each company. I want to have an alert when the query time for a client is ?% over some median value from the past, and where I am comparing Monday's data against previous Mondays' data(I am being difficult for demonstration purpose).

I want to avoid defining a separate alert for each company client.

I want to avoid defining a separate alert for each kind of query I want to monitor.

I want to avoid having each source pre-push data to one central place.

I would like to monitor some other things that has nothing to do with databases, and want to apply some entirely different logic for my alerts - e.g. some REST api call to some other system, or some ELK or time series data, or some hardware monitoring data.

I am contemplating a custom solution, but want to see if there is already a solution out there that can meet some or more of my needs. I have seen/used some monitoring solutions out there such as Nagios, though not an expert on those. Will appreciate any pointers or recommendation.

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.

Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.

It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible.

StackExchange DevOps Q#1869, answer score: 6