Service Discovery Endpoints for Prometheus and similar tools
d
deminetix
It would be great if Render had some kind of Service Discovery so that we can use monitoring tools like Prometheus
Log In
F
Fergal Armstrong
I made the mistake of trying to set up prometheus using render service discovery, only to realize all the instances were running behind the same IP, so prometheus couldn't collect them properly. I'm sure there are smarter ways of doing it, but it would be easy to setup prometheus if each instance of a service had its own IP.
Anurag Goel
Render already has service discovery built in: every service gets an internal URL and IP. Are you running into a specific issue with Prometheus?
d
deminetix
Anurag Goel: Is SD private? Where should Prometheus be getting a list of these services and their address?
Anurag Goel
deminetix: If you click on an application in the dashboard, you'll see its internal address. If the application exposes a Prometheus /metrics endpoint you can use the private URL which looks like http://your-app-slug:8080/metrics in your Prom config.
d
deminetix
Anurag Goel: Sorry I should have been clear, I meant automatic discovery. What’s the point of Prometheus monitoring your systems if it is prone to human error and doesn’t automatically update with auto scaling :P
Anurag Goel
deminetix: that's fair. We'll think about how to create Render's service discovery config for Prometheus.
E
Eduard Iskandarov
Anurag Goel: Render exposes list of services with "List services" API.
The missing part here is service instances information. API does not expose that(only
numInstances
).Once Service Instances API is exposed - it would be easy to create a HTTP-based Prometheus Service Discovery.
The instances API must return IP addresses of instances behind the service.
This should be relatively easy to implement on Render.com site, on the other hand, it will allow building more reliable applications on top of render.