How can I autoscale instances based on Kafka topic lag?

I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.

Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?

Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.

CloudWatch is absolutely the way to go on this front, you can surface Kafka Metrics in CloudWatch either by:

• Having a separate process that pushes the metrics into CloudWatch.

• Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.

The process you follow will broadly be broken down into:

• Publish Custom Metrics.

• Create an AutoScaling Launch Configuration.

• Create an AutoScaling Group and Policies based upon your Custom Metric.

Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use Spot Instances as part of your autoscaling configuration.

StackExchange DevOps Q#769, answer score: 7