๐Ÿ“ŠMonitoring

Monitoring the health of your operator setup.

Monitoring, Metrics, Grafana dashboards

Quickstart

Here you'll find a quick start guide to run the Prometheus, Grafana, and Node exporter stack. Check out the README here for more details. If you want to manually set this up, follow the steps below.

Metrics

To check if the metrics are being emitted, run the following command: Replace the EO_PROMETHEUS_PORT with the value of EO_PROMETHEUS_PORT from the data-validator/.env

curl http://localhost:<EO_PROMETHEUS_PORT>/metrics

You should see something like:

# HELP eigen_performance_score The performance metric is a score between 0 and 100 and each developer can define their own way of calculating the score. The score is calculated based on the performance of the Node and the performance of the backing services.
# TYPE eigen_performance_score gauge
eigen_performance_score{avs_name="EoracleDataValidator"} 100
...
# HELP eoracle_health_check EOracle Health Check
# TYPE eoracle_health_check gauge
eoracle_health_check{avs_name="EoracleDataValidator", name="service"} 1
eoracle_health_check{avs_name="EoracleDataValidator", name="polygon.io"} 1
...

Setup the monitoring stack

We use prometheus to scrape the metrics from the eOracle data validator container. Make sure to edit the prometheus.yml file, located at Eoracle-operator-setup/data-validator/monitoring, replacing the placeholder 'PROMETHEUS_PORT' with the actual value specified in the data validator .env file (PROMETHEUS_PORT)

cd Eoracle-operator-setup/data-validator/monitoring

The relevant lines are:

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 1m
    static_configs:
      - targets: ['eoracle-data-validator:<EO_PROMETHEUS_PORT>']

Start the monitoring stack

You can start all the monitoring stack, Prometheus, Grafana, and Node exporter all at once or only specific component

docker compose up -d

Connect docker networks

Since the eOracle data validator is running in a different docker network, we will need to have the Prometheus container in the same network of oracle-data-validator. To do that, run the following command.

docker network connect eoracle-data-validator prometheus

Troubleshooting

  • If you see the following error:

    permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/json": dial unix /var/run/docker.sock: connect: permission denied

    Use the same command by prepending sudo in front of it.

Grafana

We use Grafana to visualize the metrics from the eOracle AVS.

You can use OSS Grafana for it or any other Dashboard provider.

You should be able to navigate to http://<ip>:3000 and log in with admin/admin. This container of Grafana has a Prometheus datasource setup using port 9090. If you change the Prometheus port, you need to add a new data source or update the existing data source. You can do this by navigating to http://<ip>:3000/datasources

Useful Dashboards

We also provide a set of useful Grafana dashboards for monitoring the eOracle data validator service. You can find them here. Once you have Grafana set up, feel free to import the dashboards.

Node exporter

The eOracle data validator emits eOracle specific metrics. However, it's also important to keep track of the node's health. For this, we will use Node Exporter which is a Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written in Go with pluggable metric collectors. By default, it is installed and started when you start the entire monitoring stack. If you want to modify the stack, you can install the binary or use docker to run it.

In Grafana dashboards screen, import the node-exporter to see host metrics.

Last updated