Create Eval

Create the structure of an evaluation that can be used to test a model's performance. An evaluation is a set of testing criteria and the config for a data source, which dictates the schema of the data used in the evaluation. After creating an evaluation, you can run it on different models and model parameters. We support several types of graders and datasources. For more information, see the Evals guide.

Options

Body

The JSON request body. Pre-filled with an example which is derived from the schema.

Validate JSON

Checks “Body” against the JSON schema of the API. This happens on demand only, i.e. a failed check does not prevent the node from being executed — ultimately, the API itself decides what it accepts.

Body: Validation

Result of the last “Validate JSON” run, i.e. a confirmation or the list of schema violations. Not saved with the node settings.

Result Format

Specify how the response should be mapped to the table output. The following formats are available:

Structured Table: Returns a parsed table with data split into rows and columns.

• Object: The object type.
• Id: Unique identifier for the evaluation.
• Name: The name of the evaluation.
• Data Source Config: Configuration of data sources used in runs of the evaluation.
• Testing Criteria: A list of testing criteria.
• Created At: The Unix timestamp (in seconds) for when the eval was created.
• Metadata:

Raw Response: Returns the raw response in a single row with the following columns:

• body: Response body
• status: HTTP status code

Views

This node has no views

Workflows

• No workflows found

Links

• No links available

Developers