simple zero-shot eval function: generation length

[norakassner]

[chkla]

For a simple evaluation without defined categories for length, we can add a percentage threshold and check if the generated text is in a certain range. Possible implementation:

def eval_generation_length(generated_text, expected_len, threshold):
	"""
	Our goal is to evaluate the length of a given generated text string in terms of expected and actual length with a threshold.

	Example: 
		Prompt: "a text with around [100] chars" expected length 100 chars, 
		Generated text: "the cake is a lie ..." actual text length 93,
		Evaluation with threshold 10%: check if text 93 is between 100-10% and 100+10% 
	"""

	input_len = len(generated_text)
	threshold_range = expected_len*(threshold/100)

	expected_len_min = expected_len-threshold_range # len - threshold
	expected_len_max = expected_len+threshold_range # len + threshold

	print("For a given text with {} chars, the threshold {} is used to define a soft boundary to check if it is between {} and {}".format(input_len, threshold, expected_len_min, input_len, expected_len_max))

	if expected_len_min < input_len and input_len < expected_len_max:
		return True
	else:
		return False

if __name__ == "__main__":
	text = "Obama was the guest of honor at the conference. Bieber performed at the concert last night." # generated text
	prompt_defined_len = 100 # text with x chars
	threshold = 25 # x %

	print("Evaluation: {}".format(eval_generation_length(text, prompt_defined_len, threshold)))