Connect CosmicAC to your Prometheus and Loki

Connect CosmicAC to your Prometheus and Loki so job pages show charts and log history.

Connect CosmicAC to your Prometheus and Loki so job pages show charts and log history. Job metrics go to Prometheus, and job logs go to Loki.

You run and maintain both, including their storage, retention, and access control. CosmicAC doesn't install or operate either one. To install them, see the Prometheus installation guide and the Loki installation guide.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
A running Prometheus and a running Loki.
The host and port that cosmicac-wrk-monitor serves on. It listens on port 9110 unless your deployment sets another.

Allow these three network paths between your Prometheus, your Loki, and cosmicac-wrk-monitor.

Allow	Without it
Your Prometheus to reach `cosmicac-wrk-monitor` on port `9110`	Prometheus scrapes nothing, and every chart stays empty
`cosmicac-wrk-monitor` to reach your Loki	Log history stays empty, though live tail still works
`cosmicac-wrk-monitor` to reach your Prometheus	The charts on a job page stay empty, even while Prometheus scrapes normally

CosmicAC connects to your Prometheus and Loki at the URLs you save in steps 2 and 3. Include a port if yours uses one.

Steps

Open the Observability page

In the left navigation, click Settings. On the Settings page, under Instance, click Observability.

Connect Prometheus

In the Prometheus query URL field, enter the base URL of your Prometheus, for example http://prometheus:9090. Click Save.

CosmicAC reads from this URL to draw the charts on a job page. It never writes to your Prometheus. Step 4 sets up the opposite direction, where your Prometheus scrapes metrics from CosmicAC.

Connect Loki

In the Loki query URL field, enter the base URL of your Loki, for example http://loki:3100. Click Save.

CosmicAC doesn't send Loki's X-Scope-OrgID tenant header. On a single-tenant Loki, set auth_enabled: false. On a multi-tenant Loki, add a proxy in front that sets the header.

Add CosmicAC as a Prometheus scrape target

In your Prometheus configuration, add cosmicac-wrk-monitor as a scrape target:

scrape_configs:
  - job_name: cosmicac-jobs
    scrape_interval: 5s
    honor_labels: true
    # Remove these three lines unless your deployment sets a scrape token.
    authorization:
      type: Bearer
      credentials: '<scrape-token>'
    static_configs:
      - targets: ['<monitor-host>:9110']

Remove the authorization lines unless your deployment sets metricsScrapeToken. If it does, replace <scrape-token> with that value. A scrape that omits a required token fails with 401.

honor_labels must be on

Without honor_labels: true, Prometheus overwrites the job ID in the instance label, so the CPU and memory charts show no data. The GPU charts still work, because GPU series use the job_id label.

A scrape stores one value per series, so scrape_interval sets how much detail the charts show. Job agents sample about once a second, so the 5s above keeps one sample in every five. A lower interval keeps more samples and stores more data.

Confirm the connection

A card reports Connected as soon as it has a saved URL. That status doesn't confirm that CosmicAC reached Prometheus or Loki.

To confirm that Prometheus scrapes CosmicAC, open Status, then Targets in the Prometheus web interface. The cosmicac-jobs target reports UP.

To confirm that CosmicAC queries Prometheus, request the metrics for a running job:

curl http://<monitor-host>:9110/job-metrics/<job-id>

The response carries cpu and gpu values. Wait about 5 seconds after a job starts, so Prometheus scrapes at least once first.

Restrict access to `cosmicac-wrk-monitor`

On cosmicac-wrk-monitor, the /logs and /logs/history endpoints require no authentication. /metrics is open too, unless your deployment sets a scrape token. Allow only trusted networks to reach cosmicac-wrk-monitor, and add authentication before you expose it.

To require a token on /metrics, set metricsScrapeToken in the cosmicac-wrk-monitor configuration. Then keep the authorization lines in your scrape job, as shown in step 4.

Disconnect Prometheus or Loki

To disconnect one, click Disconnect on its card, then confirm. CosmicAC clears that URL and leaves the other in place.

To disconnect both, click Disconnect all at the top of the page, then confirm.

Disconnecting Loki stops CosmicAC from pushing new log lines. Disconnecting Prometheus doesn't stop your Prometheus from scraping. To stop the scrape, remove the target from your Prometheus configuration.

Help and troubleshooting

The target reports UP but no job series arrive

Run a job. Series exist only while a job runs, and they leave the target about a minute after it ends.

GPU charts populate but CPU and memory charts stay empty

Add honor_labels: true to your scrape job, then reload Prometheus.

The charts show fewer points than you expect

Lower scrape_interval. Prometheus stores one value per series per scrape.

The charts fail with a 503 ERR_PROMETHEUS_NOT_CONFIGURED error

Save a Prometheus query URL in step 2, and allow cosmicac-wrk-monitor to reach your Prometheus.

Log history fails with a 503 ERR_LOKI_NOT_CONFIGURED error

Save a Loki query URL in step 3. Live tail keeps working without it.

Log history fails with a 400 ERR_LOKI_QUERY_REJECTED error

Narrow the time range, or raise max_query_length in your Loki configuration.

Log history fails with a 502 ERR_LOKI_UNAVAILABLE error

Check that the saved URL is the root that serves /loki/api/v1/. If Loki sits behind a proxy that adds a path prefix, include the prefix.

Scraping fails with a 401 error

Keep the authorization lines in your scrape job, as shown in step 4.

A finished job's charts are empty

Query Prometheus directly over a time range that covers when the job ran.