evals
evals copied to clipboard
openai
Legal Ethics - Model Rules of Professional Conduct - True/False
Thank you for contributing an eval! ♥️
🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨
PLEASE READ THIS:
In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.
We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.
Eval details 📑
Eval name
law-true-false
Eval description
The file contains a series of 102 True/False questions. The questions are based on the American Bar Association's Model Rules of Professional Conduct, which are the model legal ethics rules for lawyers.
What makes this a useful eval?
At some point, someone is going to try to build tools to help legal professionals based on GPT-4. Although it would be completely insufficient to safeguard against unethical use on its own, as a baseline, I think there's a lot of value in understanding how well the model can currently "understand" legal ethics issues. Failure to perform well on these evals might indicate areas in which the model could benefit from being fine-tuned. Beyond that, I (and probably others) am curious how well the model currently performs here. These rules aren't binding in any specific jurisdiction, but many jurisdictions model their rules after them.
NOTE: I'm a law student, not a lawyer. All of the true/false designations are correct to the best of my knowledge, but it's possible there is room for improvement on the questions to make them clearer and/or more valid.
GPT-3 scored a 73% on this set. I can probably make these questions harder if GPT-4 is doing significantly better, but I started off with questions that basically anyone could answer just by reading through the rules. I even had ChatGPT generate some of them based on the rules, and then I spot-checked them and changed some of the wording. In general, I'm committed to adding on to this over time to add more coverage of the rules and also increase the complexity of the questions. It would be interesting to find people who are better versed in the subject matter to collaborate with as well.
Here is just one example of a useful insight from this exercise (at least as it pertains to GPT-3). I ask it the following question:
"If a client does not respond within 60 days after receiving notice of the proposed sale of their lawyer's practice, their consent to the transfer of their files is presumed."
And in the response (which I can see in the results .jsonl file), I can see that it says the answer is "True", and it provides the following reason:
"According to Rule 1.17(b) of the ABA Model Rules of Professional Conduct, if a client does not object to the proposed sale of their lawyer's practice within 60 days after receiving notice, their consent to the transfer of their files is presumed"
However, you can see for yourself that both (1) the citation and (2) the number of days are provided very confidently (but they are incorrect). The relevant section is really 1.17(c)(3), and it's really 90 days. This demonstrates that it may not be very effective with numbers in a legal context, which could lead to problematic outcomes. And it may be prone to suggestion considering the "60 day" number has no basis in reality and was simply included in the question.
I also included the exact same question, but with the (correct) number of 90 days. In that case, it said "true" (which was correct). But it shows the model can simultaneously say two completely contradictory things, and what it's saying is more likely to mirror the form of the question.
Update - March 15
To test the "numerical suggestion" hypothesis further, I added several other true/false questions of the following form:
"A lawyer should aspire to render at least [10/25/50/100] hours of pro bono legal services each year."
As expected, it said "True" to all of them, and when it provided its reasoning, it said that the rules specifically state that the number should be XX hours. Although the only correct answer is 50. So this is additional evidence of strong suggestion bias, especially as it relates to numbers.
This is particularly relevant in a legal setting, where exact numbers can completely determine the outcome of a case. Such as if you miss the filing deadline ("N days"), you miss the amount in controversy in a federal diversity filing (the federal rules require that the amount exceeds $75K), etc. This is perhaps another class of problem that can be explored further in a separate eval. But I think it's interesting to include here, and there is value in having the more general true/false benchmark too.
Criteria for a good eval ✅
Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).
Your eval should be:
- [x] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
- [x] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
- [x] Includes good signal around what is the right behavior. This means either a correct answer for
Basicevals or theFactModel-graded eval, or an exhaustive rubric for evaluating answers for theCriteriaModel-graded eval. - [x] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)
If there is anything else that makes your eval worth including, please document it below.
Unique eval value
Insert what makes your eval high quality that was not mentioned above. (Not required)
Eval structure 🏗️
Your eval should
- [x] Check that your data is in
evals/registry/data/{name} - [x] Check that your yaml is registered at
evals/registry/evals/{name}.jsonl - [x] Ensure you have the right to use the data you submit via this eval
(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)
Final checklist 👀
Submission agreement
By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).
- [x] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.
Email address validation
If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.
- [x] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.
Limited availability acknowledgement
We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.
- [x] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.
Submit eval
- [x] I have filled out all required fields in the evals PR form
- [x] (Ignore if not submitting code) I have run
pip install pre-commit; pre-commit installand have verified thatblack,isort, andautoflakeare running when I commit and push
Failure to fill out all required fields will result in the PR being closed.
Eval JSON data
Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:
View evals in JSON
Eval
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer with general experience not considered competent to handle a case involving a specialized field of law."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Newly admitted lawyers cannot be as competent as practitioners with long experience."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In an emergency, a lawyer is allowed to provide assistance in a matter even if they do not have the ordinarily required skill."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must always prepare for legal problems with which they are not familiar by seeking special training."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer cannot accept representation if they do not currently possess the required level of competence."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney may consult with lawyers outside of the attorney's firm on the client's case without telling the client."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney may consult with lawyers inside of the attorney's firm on the client's case without telling the client."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When lawyers from more than one law firm are providing legal services to a client on a particular matter, they do not need to consult with each other and the client about the scope of their respective representations."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Maintaining competence does not require a lawyer to keep abreast of changes in the law and its practice."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer cannot ethically represent a client if they disagree with the client's political, economic, social or moral views related to the matter."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may limit the scope of representation if it is reasonable and the client gives informed consent."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In a criminal case, the lawyer has the final say in decisions regarding the client's plea, whether to waive jury trial, and whether the client will testify."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can counsel a client to engage in conduct that the lawyer knows is criminal or fraudulent."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can counsel a client regarding the consequences of conduct that the lawyer knows is criminal or fraudulent."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer's representation of a client does not imply endorsement of the client's views or activities."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to draft and deliver documents that they know are fraudulent in order to assist their client."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When a lawyer discovers that their client's ongoing conduct is criminal or fraudulent, the lawyer must report the client to the authorities."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may provide limited representation to a client as long as the limitation is reasonable and the client is properly informed."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is prohibited from discussing the legal consequences of a proposed course of conduct with a client if that conduct is clearly criminal."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "If a lawyer knows that a client expects assistance not permitted by the Rules of Professional Conduct, the lawyer must immediately withdraw from the representation."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to charge a contingent fee in a domestic relations matter, where the payment is contingent upon the securing of a divorce or the amount of alimony or support, when the client is unable to pay otherwise."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must communicate the scope of representation and the basis or rate of the fee and expenses to the client in writing before the representation commences in all cases."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may charge a client an unreasonable fee or an unreasonable amount for expenses, as long as the client agrees to it."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to enter into a fee arrangement that might induce the lawyer to curtail services for the client or perform them in a way contrary to the client's interest."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A contingent fee agreement must be orally agreed upon between the lawyer and client, stating the method by which the fee is to be determined."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may accept property in payment for services, such as an ownership interest in an enterprise, without any limitations or restrictions."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may divide a fee with another lawyer who is not in the same firm, even if the client does not agree to the arrangement."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is required to return any unearned portion of a fee if the client fires them without warning."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may charge a contingent fee for representing a defendant in a criminal case."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When determining the reasonableness of a fee, factors such as the time and labor required, the novelty and difficulty of the questions involved, and the skill requisite to perform the legal service properly should be considered."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to reveal information relating to the representation of a client if they believe it is necessary to prevent the client from committing a crime.."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to reveal information relating to the representation of a client if they believe it is necessary to prevent the client from committing a crime that may result in financial injury to another person."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is required to reveal information relating to the representation of a client if it is necessary to prevent the client from committing a crime that may result in financial injury to another person."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may reveal information relating to the representation of a client to detect and resolve conflicts of interest arising from changes in the composition or ownership of a firm, even if the revealed information would compromise the attorney-client privilege."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may disclose information relating to the representation of a client if the disclosure is impliedly authorized to carry out the representation."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is prohibited from revealing any information relating to the representation of a client, regardless of the circumstances."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may disclose information relating to the representation of a client to secure legal advice about the lawyer's compliance with the Rules of Professional Conduct."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to reveal information relating to the representation of a client to establish a defense in a controversy between the lawyer and the client."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must disclose unprotected information relating to the representation of a client, even if the disclosure could reasonably lead to the discovery of protected information by a third party."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to disclose information relating to a client's representation to detect and resolve conflicts of interest without the client's consent."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must take reasonable efforts to prevent unauthorized access to or disclosure of client information, even if the lawyer is not at fault for the unauthorized access or disclosure."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is not required to take any special security measures when transmitting client information if the method of communication affords a reasonable expectation of privacy."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "The duty of confidentiality ends when the client-lawyer relationship is terminated."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must always appeal an adverse ruling regarding disclosure of client information, regardless of the client's wishes."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to disclose client information if a court or other tribunal orders the disclosure, even without the client's informed consent."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may represent a client with a concurrent conflict of interest without obtaining informed consent from each affected client, as long as the lawyer reasonably believes they can provide competent and diligent representation."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is allowed to represent a client if the representation involves asserting a claim by one client against another client represented by the same lawyer in the same litigation."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may represent clients with economically adverse interests in unrelated matters without obtaining their informed consent."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer's duty of loyalty and independent judgment can be materially limited by responsibilities to a former client."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is permitted to represent a client in a transaction where the lawyer's own conduct is in question, as long as the lawyer can provide competent and diligent representation."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may engage in sexual relationships with a client if the relationship predates the formation of the client-lawyer relationship."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who is closely related by blood or marriage to another lawyer may represent a client in a matter where the other lawyer represents another party, without obtaining informed consent from each client."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may accept payment from a source other than the client if the client is informed and consents, and the arrangement does not compromise the lawyer's duty of loyalty or independent judgment."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A parent who pays for the lawyer's services on behalf of a child may direct or control some of the legal strategy decisions made."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "In all cases, clients can consent to a representation where a conflict of interest exists."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "Representation is prohibited if the lawyer cannot reasonably conclude that they will be able to provide competent and diligent representation."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "Informed consent requires the client to be aware of the possible effects on loyalty, confidentiality, and the attorney-client privilege when multiple clients are represented in a single matter."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A client who has given consent to a conflict cannot revoke the consent or terminate the lawyer's representation at any time."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A client who has given consent to a conflict cannot revoke the consent or terminate the lawyer's representation if the lawyer reasonably and honestly believes the timing would harm the client's interests."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "General and open-ended advance consent to future conflicts is considered effective."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A lawyer may not take inconsistent legal positions in different tribunals at different times on behalf of different clients."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the information provided about conflict of interest."}, {"role": "user", "content": "A lawyer is required to obtain informed consent from a client, confirmed in writing, when there is a potential conflict of interest."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer is prohibited from representing multiple parties to a negotiation if their interests are fundamentally antagonistic to each other."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can represent multiple clients with generally aligned interests even if there are some differences in interest among them."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must always maintain impartiality between commonly represented clients."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "As between commonly represented clients, the attorney-client privilege does not attach."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer representing an organization also represents all of its affiliated organizations, such as parent and subsidiary companies."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who is a member of a corporation's board of directors must resign from the board or cease acting as the corporation's lawyer when a conflict of interest arises."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In common representation, if one client asks the lawyer not to disclose information relevant to the representation to the other client, the lawyer must withdraw from representing both clients."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer representing multiple clients in the same matter should consider the potential additional cost, embarrassment, and recrimination if the common representation fails."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "When seeking to establish or adjust a relationship between clients, the lawyer's role is that of partisanship normally expected in other circumstances."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In a law firm, a lawyer's disqualification due to a personal interest will result in the disqualification of all other lawyers in the firm."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who moves from one firm to another can be screened from participation in a matter, and the new firm can represent a client with adverse interests without obtaining the former client's informed consent."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A law firm is prohibited from representing a client with interests adverse to those of a client represented by a formerly associated lawyer."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A disqualification prescribed by Rule 1.10 may be waived by the affected client under the conditions stated in Rule 1.7."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer's disqualification based on prior work as a law student will result in the disqualification of all other lawyers in the firm."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney is allowed to touch and move contraband on behalf of the client."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "In ex parte proceedings, an attorney is not required to reveal information that may be harmful to their client's case."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A prosecutor is obligated to timely disclose favorable evidence to the defense, even if it is inadmissible or has no impact on the outcome of the case."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is acceptable for an attorney to communicate directly with a person who is represented by counsel on a specific matter without the consent of their counsel."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A prosecutor has a duty to protect the accused's right to counsel."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer can make false statements of fact to adversaries and third parties."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney must not act with the sole purpose of delaying, burdening, or embarrassing other parties while obtaining evidence."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Lawyers are allowed to make out-of-court statements that are completely true, but that they reasonably should know have a substantial likelihood of materially prejudicing the case."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is acceptable for a prosecutor to make true comments that have a substantial likelihood of heightening public condemnation of the accused."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer must self-report when they know they have violated the rules of professional conduct."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Attorneys have an affirmative duty to expedite cases and should not delay cases for their own personal gain or convenience."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A licensed attorney may practice law in a jurisdiction where they are not licensed in an emergency situation if it can avoid a substantial injustice."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney must report any attorney or judge's violation of the Rules if it raises a substantial question as to their honesty, trustworthiness, or fitness as a lawyer."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A subordinate lawyer who follows an order to take an action in violation of the Rules is not subject to discipline if the ethical responsibility is debatable."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney has no duty to follow valid procedural rules and court orders that they reasonably believe are defective or invalid."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "Attorneys must not engage in conduct involving dishonesty, fraud, deceit, or misrepresentation, even in their private business or personal life that is unrelated to the practice of law."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is permissible for an attorney to talk to members of the jury before or during a trial, so long as the discussion is not about the trial."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "An attorney may be disciplined for failing to prevent ethical violations of other members of their law firm."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer may sell certain cases of their law practice and retain others, depending on how valuable they are."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who sells their entire practice may subsequently work as in-house counsel for a business."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "During the sale of a law practice, the seller must obtain client consent before sharing detailed information about a client's case with the potential buyer."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "If a client does not respond within 60 days after receiving notice of the proposed sale of their lawyer's practice, their consent to the transfer of their files is presumed."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "If a client does not respond within 90 days after receiving notice of the proposed sale of their lawyer's practice, their consent to the transfer of their files is presumed."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "It is a violation of the ABA Model Rules of Professional Conduct for a lawyer to sell only a specific area of their law practice."}], "ideal": "False"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer who sells an area of their practice must cease accepting any matters in that area, including as counsel or co-counsel, after the sale."}], "ideal": "True"}
{"input": [{"role": "system", "content": "You are LawStudentGPT. Answer the following True/False question according to the ABA Model Rules of Professional Conduct."}, {"role": "user", "content": "A lawyer selling their practice is allowed to share client confidences with the potential buyer without client consent during preliminary negotiations."}], "ideal": "True"}
@avery-bub,
Thanks for making an eval for MPRC rules.
I thought you might be curious about the accuracy of GPT-4 with your eval:
GPT-4 ACCURACY: 0.8317757009345794
(env) user@user:/tmp/evals$ oaieval gpt-4 law_true_false.dev.v0
[2023-03-16 22:59:30,759] [registry.py:145] Loading registry from /tmp/evals/evals/registry/evals
[2023-03-16 22:59:30,770] [registry.py:145] Loading registry from /home/user/.evals/evals
[2023-03-16 22:59:31,258] [oaieval.py:178] Run started: 230317025931VZQ3BBST
[2023-03-16 22:59:31,260] [data.py:78] Fetching law_true_false/samples.jsonl
[2023-03-16 22:59:31,261] [eval.py:30] Evaluating 107 samples
[2023-03-16 22:59:31,269] [eval.py:136] Running in threaded mode with 10 threads!
[2023-03-16 23:00:22,191] [record.py:320] Final report: {'accuracy': 0.8317757009345794}. Logged to /tmp/evallogs/230317025931VZQ3BBST_gpt-4_law_true_false.dev.v0.jsonl
[2023-03-16 23:00:22,192] [oaieval.py:209] Final report:
[2023-03-16 23:00:22,192] [oaieval.py:211] accuracy: 0.8317757009345794
[2023-03-16 23:00:22,193] [record.py:309] Logged 21 rows of events to /tmp/evallogs/230317025931VZQ3BBST_gpt-4_law_true_false.dev.v0.jsonl: insert_time=0.943ms
The next class of question I tried consists of questions that state rules in a very specific way with specific numbers associated with them, but the rule is completely made up. In these cases, the model always says the statements are "True", even though there is no basis in reality for the statements. It seems that the model is more prone to saying "True" when a rule is worded in a very official way even if part of it clearly deviates. For example, there is one rule:
"a lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless all parties to the proceeding give informed consent, confirmed in writing."
That I changed to:
"A lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless 10 years have passed after the laywer's last point of involvement."
It does not pick up on the fact that this statement is false (at least for GPT-3.5)
To nie chodzi o twoją interpretacje i nie ma nic wspólnego z wątkiem który podałeś nie wiem nawet o którym fragmencie części mówisz moja wymiana informacji z GTP nie ma nic wspólnego z tym co robicie i nie mam nawet zielonego pojecia co te wasze szlaczki znaczą . Większość jest losowym zbiegiem okoliczności i powstało z waszej strony. mi to tylko przeszkadza raz coś niespecjalnie uruchomiłem i zasypało mnie tym szajsem potem coś żeby komuś nie namieszać potwierdzałem jak mi się wydaje wez to sobie jakoś jak wam potrzebne o wpisz gdzie są żebym niepsial jakimś idiotom odpisywać ma coś czego nie rozumie
sob., 18 mar 2023, 17:18 użytkownik Avery Bub @.***> napisał:
The next class of question I tried consists of questions that state rules in a very specific way with specific numbers associated with them, but the rule is completely made up. In these cases, the model always says the statements are "True", even though there is no basis in reality for the statements. It seems that the model is more prone to saying "True" when a rule is worded in a very official way even if part of it clearly deviates. For example, there is one rule:
"a lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless all parties to the proceeding give informed consent, confirmed in writing."
That I changed to:
"A lawyer shall not represent anyone in connection with a matter in which the lawyer participated personally and substantially as a judge or other adjudicative officer or law clerk to such a person or as an arbitrator, mediator or other third-party neutral, unless 10 years have passed after the laywer's last point of involvement."
It does not pick up on the fact that this statement is false (at least for GPT-3.5)
— Reply to this email directly, view it on GitHub https://github.com/openai/evals/pull/95#issuecomment-1474891931, or unsubscribe https://github.com/notifications/unsubscribe-auth/A6PJHHVHEXF4C6VG6H3EKATW4XN4DANCNFSM6AAAAAAV3HHO2M . You are receiving this because you are subscribed to this thread.Message ID: @.***>
@andrew-openai
Could you please take a look at the user @SierotkaM, the comments which he adds to issues and pull requests all over this repository are not helpful or related in any form, and it seems like he has subscribed to e-mail notifications and now doesn't know how to unsubscribe, although I already pointed to the respective GitHub docs.
