API Health Monitor

Continuously monitor 74+ insurance partner APIs and surface their live status on a dashboard, instead of finding out about a partner outage from a support ticket.

Checking 74+ endpoints on a schedule without overwhelming them or taking too long to notice a real outage required genuine concurrency, plus a classification scheme that distinguishes a truly down partner service from one that is just responding slowly — and alerting needed to fire on real state changes without drowning the team in noise.

An async scheduler drives a concurrent health checker (Python asyncio + aiohttp), bounded by connection limits so it does not hammer any single partner, against an API list loaded dynamically from configuration. Each result is classified as UP, SLOW, DOWN, or ERROR based on status code and response time, feeding a multi-channel alerting layer (email, Slack, Teams) with smart state-change suppression, and two live dashboards — a FastAPI/SSE view and a Streamlit + Plotly view.

The API list is entirely config-driven — adding a new partner endpoint to monitor is a configuration change, not a code change. Per-API isolation, configurable timeouts, and exponential backoff retries keep one struggling partner from cascading into false alerts for everyone else, and state-change suppression means alerts fire on a real transition rather than on every failed check.

Runs as a Docker Compose stack with its own PostgreSQL instance for history, Prometheus metrics for the underlying monitor's own health, and a daily retention-cleanup job so history does not grow unbounded.

One failing partner API check should not take down the monitor itself, delay checks on every other partner, or trigger an alert storm.

Every check runs isolated inside its own error handling within the concurrent batch, exponential backoff smooths over transient blips before they escalate, and alerts only fire on an actual status-change transition rather than on every individual failed check.

Validated operationally against 74+ real partner endpoints in continuous use; an automated regression test suite is a documented next step.

Rounding out the alerting integrations further and adding an automated regression test suite are the next steps.