Convert the evaluation data into formats that can be used by the evaluator. This should most commonly be a string. Parameters are the raw input from the run, the raw output, raw reference output, and the raw run.
// Chain input: { input: "some string" }
// Chain output: { output: "some output" }
// Reference example output format: { output: "some reference output" }
const formatEvaluatorInputs = ({
rawInput,
rawPrediction,
rawReferenceOutput,
}) => {
return {
input: rawInput.input,
prediction: rawPrediction.output,
reference: rawReferenceOutput.output,
};
};
The prepared data.
OptionalagentA list of tools available to the agent, for TrajectoryEvalChain.
OptionalchainOptionalcriteriaThe criteria to use for the evaluator.
OptionaldistanceThe distance metric to use for comparing the embeddings.
OptionalembeddingThe embedding objects to vectorize the outputs.
OptionalfeedbackThe feedback (or metric) name to use for the logged evaluation results. If none provided, we default to the evaluationName.
Optionalllm
The name of the evaluator to use. Example: labeled_criteria, criteria, etc.