Skip to main content

Metrics

A simulation wouldn't be worth doing if it's just a black box that doesn't output anything. The main method for outputting in V1 is Metrics. The Metric system has been designed with a few prebuilt methods that can be turned on and configured to collect at certain steps. Most classes and functions expose a metric_collector to you which you can use to collect metrics.

This class can be used to collect metrics using the collect function. This function takes two arguments:

  • metric_name which is the name of the metric that you're trying to log.
  • arguments which is an object of arguments.
metric_collector.collect('balance', {'value': 100000, 'token': 'ETH', 'address': '0x6429FEe053768Ffa90a59cAfb98Ca9E8F6471211'})

This will add a structured log entry to the log file which will available for you to download at the end of the simulation. The log entry when calling it like this in your custom code will automatically populate some other fields about the current environment step and when the metric was collected. The amount of arguments here is unlimited, feel free to add as much data as you'd like.

Metrics are processed asynchronously in a Queue. If there's an issue with the metrics being processed the simulator will wait for 30 seconds after the simulation to see if the amount of yet to process metrics decreases. If it doesn't it will throw a Timeout error and mark the simulation as Failed (even though there was a result).

Prebuild​

We currently support a setting that will automatically log metrics for you.

  • collectTransactionMetrics when set to True will log information about every transaction sent to the chain during the simulation after a block is mined. It will include information about the inputs, and if possible try to decode the transaction if the contract for it is available. If the transaction was successful it will also include the results from the transaction.

Class implementation​

Each of the 3 metric types currently available (Agent, Environment and Simulation) have to implement separate classes with separate almanak-library.yaml files.

Library file​

The library file for the protocol supports a few extra parameters besides the standard ones, see a full example of a library file for an agent below:

name: "Amazing Environment Metric"
version: "1.0.0"
type: "metric"
metric_type: "environment"
author: "Almanak AG"
description: ""
license: "MIT"

settings:
  example_setting: "example_value"

dependencies:
  - "library://almanak/protocols/aave-v3:1.0.0"

Please note that only 1 almanak-library.yaml file is required per protocol, each protocol can implement 1 or all 3 types of protocol.

As you can see the main part that we add is the supported_chains, which is simply an array of objects that you can use to configure dependencies and only allow the agent to work on certain chains because it has unique features to them.

  • metric_type has to be agent, environment or global respectively.
  • dependencies allow you to set certain dependencies on an environment level, these have to be library items.
  • settings are like any other settings but can overwrite them on an environment level. Useful for contract addresses that are different per chain
Tutorial

See the Results & Metrics Tutorial to learn how to obtain and analyze the results and metrics.