Fibonacci word selection character count total

[mstooks]

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.

Eval details 📑

Eval name

fibonacci-letter-count

Eval description

Make a sentence into an array. Use the Fibonacci sequence to pick the corresponding words from the array. Get the character counts of the selected words and sum them.

(Count the letters in a sentence using the Fibonacci sequence to pick words. Ignore punctuation and symbols. Give the sum as an integer.)

What makes this a useful eval?

The model knows very well what the Fibonacci sequence is, it demonstrates the model's ability to understand complex instructions, perform mathematical calculations, and manipulate text data. Successfully completing this task showcases the model's versatility, and its potential for NLP and computational tasks.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• [x] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• [x] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• [x] Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• [x] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

The model was able to produce code which could execute this task successfully but was unable to do the task it's self, this shows an area of weakness in reasoning when interpreting the raw text it's self vs trying to solve it programatically.

Eval structure 🏗️

Your eval should

• [x] Check that your data is in evals/registry/data/{name}
• [x] Check that your yaml is registered at evals/registry/evals/{name}.jsonl
• [x] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• [x] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• [x] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• [x] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• [x] I have filled out all required fields in the evals PR form
• [ ] (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "London was the first city in the world to have an underground railway system, which opened in 1863."}], "ideal": "30"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Moscow is the largest city in Europe by population, with over 12 million inhabitants."}], "ideal": "43"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Tokyo is the world's most populous city, with over 37 million people in its metropolitan area."}], "ideal": "33"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "New York City has more than 800 languages spoken within its boundaries, making it the most linguistically diverse city in the world."}], "ideal": "35"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Istanbul is the only city in the world that straddles two continents - Europe and Asia."}], "ideal": "28"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Paris is known as the City of Light because it was one of the first cities in the world to have street lighting."}], "ideal": "36"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Dubai is home to the world's tallest building, the Burj Khalifa, which stands at 828 meters tall."}], "ideal": "26"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Los Angeles has more cars per capita than any other city in the world."}], "ideal": "35"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Mumbai has the largest number of millionaires and billionaires in India."}], "ideal": "36"}
{"input": [{"role": "system", "content": "Please provide only the total character count of the words corresponding to Fibonacci positions starting from zero and considering large numbers as words (ignoring punctuation) in the following sentence, respond only with a number and no explanation. Do not include symbols or equations in your response (including + or =), Only reply with the numerical sum of the characters. Make sure the first word is your answer."}, {"role": "user", "content": "Rio de Janeiro is home to the world-famous Christ the Redeemer statue, which stands at 98 feet tall."}], "ideal": "30"}

[mstooks]

Thank you for this recommendation, I have gone through some iterative tests and simplified the prompt, this test now gives more accurate numerical results (most of the time). Please let me know if you have additional recommendations.

[mstooks]

Thank you for your feedback. Attached is also a verification tool for testing of these evals

#Mitchell Stooksbury - FANTASTIC (Fibonacci Algorithm Noting Total Alphabetical String Tally In Characters) Eval checker
import string

def fibonacci(n):
    """
    Returns the nth Fibonacci number.
    """
    if n < 2:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

def fibonacci_word_count(sentence):
    """
    Computes the sum of the total number of letters from the words in the sentence
    whose indices are Fibonacci numbers starting from 0. Returns the total character
    count for the selected words.
    """
    word_list = sentence.translate(str.maketrans('', '', string.punctuation)).split()
    total_letters = 0
    index = 0
    print("Words in sentence (punctuation removed): ",word_list)
    while fibonacci(index) < len(word_list):
        word_index = fibonacci(index)
        word = word_list[word_index]
        print(word)
        total_letters += len(word)
        index += 1
    
    return total_letters

print('\n Total Count:',fibonacci_word_count("Two dollar bills can be exchanged for other denominations of U.S. currency at most banks."))

## Read input file
#with open("sentences.txt", "r") as input_file:
#    lines = input_file.readlines()
#
## Compute word count for each line and write output to file
#with open("answers.txt", "w") as output_file:
#    for line in lines:
#        line = line.strip()
#        word_count = fibonacci_word_count(line)
#        output_file.write(f"{word_count}\n")

[luqman-openai]

Closing for now, please feel free to reopen if you get a chance to address the comments.