Metrics and Monitoring

Setup the monitoring stack

These instructions provide a quick start guide to run the Prometheus, Grafana, and Prometheus Node exporter stack.

Metrics

To check if the metrics are being emitted, run the following command: Replace the PROMETHEUS_PORT with the value of PROMETHEUS_PORT from the data-validator/.env

curl http://localhost:<PROMETHEUS_PORT>/metrics

You should see something like

# HELP eigen_performance_score The performance metric is a score between 0 and 100 and each developer can define their own way of calculating the score. The score is calculated based on the performance of the Node and the performance of the backing services.
# TYPE eigen_performance_score gauge
eigen_performance_score{avs_name="EoracleDataValidator"} 100
...
# HELP eoracle_health_check eoracle Health Check
# TYPE eoracle_health_check gauge
eoracle_health_check{avs_name="EoracleDataValidator", name="service"} 1
eoracle_health_check{avs_name="EoracleDataValidator", name="polygon.io"} 1

Update Prometheus configuration

We use prometheus to scrape the metrics from the eoracle data validator container.

Make sure to edit the prometheus.yml file, located at Eoracle-operator-setup/data-validator/monitoring, replacing the placeholder 'PROMETHEUS_PORT' with the actual value specified in the data-validator .env file (PROMETHEUS_PORT)

cd Eoracle-operator-setup/data-validator/monitoring

The relevant lines are:

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 1m
    static_configs:
      - targets: ['eoracle-data-validator:<PROMETHEUS_PORT>']

Start the monitoring stack

You can start all the monitoring stack, Prometheus, Grafana, and Node exporter all at once or only specific component

Start the entire monitoring stack

docker compose up -d

Connect docker networks

Since the eoracle data validator is running in a different docker network, we will need to have the Prometheus container in the same network of oracle-data-validator. To do that, run the following command

docker network connect eoracle-data-validator prometheus  

Grafana

We use Grafana to visualize the metrics from the eoracle AVS.

You can use OSS Grafana for it or any other Dashboard provider.

You should be able to navigate to http://<ip>:3000 and log in with admin/admin. This container of Grafana has a Prometheus datasource setup using port 9090. If you change the Prometheus port, you need to add a new data source or update the existing data source. You can do this by navigating to http://<ip>:3000/datasources

Useful Dashboards

We also provide a set of useful Grafana dashboards which would be useful for monitoring the eoracle data validator service. You can find them here. Once you have Grafana set up, feel free to import the dashboards.