emdgroup
Hardness benchmark
Work in progress Integrated Hardness benchmarking task.
To-do:
- replace the dataset
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 3 committers have signed the CLA.
:white_check_mark: ritalyu17
:x: pre-commit-ci[bot]
:x: Rita Lyu
Rita Lyu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.
The hardness benchmark is ready for review and some feedbacks.
Currently, the bayesian optimization component and multi-task component are set to two Benchmark. Main reason for seperating them is because the arguments in simulate_scenarios are different, specifically initial_data. Maybe there is a way to make the code look nicer?
Thank you!
Hello @ritalyu17 just for your information: My work load has shifted quite a bit, and it might take some time for me to properly review here. Just wanted to inform you about this :)
Hi Adrian, thank you for following up. I am planning on wrapping this up by end of this week.
On Tue, Feb 25, 2025 at 8:47 AM AdrianSosic @.***> wrote:
Hi @ritalyu17 https://github.com/ritalyu17, any updates from your end?
— Reply to this email directly, view it on GitHub https://github.com/emdgroup/baybe/pull/440#issuecomment-2682036759, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVQ5HTYKVSH3TG5GVTQQKCT2RRYAJAVCNFSM6AAAAABS5B4BEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBSGAZTMNZVHE . You are receiving this because you were mentioned.Message ID: @.***> [image: AdrianSosic]AdrianSosic left a comment (emdgroup/baybe#440) https://github.com/emdgroup/baybe/pull/440#issuecomment-2682036759
Hi @ritalyu17 https://github.com/ritalyu17, any updates from your end?
— Reply to this email directly, view it on GitHub https://github.com/emdgroup/baybe/pull/440#issuecomment-2682036759, or unsubscribe https://github.com/notifications/unsubscribe-auth/AVQ5HTYKVSH3TG5GVTQQKCT2RRYAJAVCNFSM6AAAAABS5B4BEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBSGAZTMNZVHE . You are receiving this because you were mentioned.Message ID: @.***>
Hi @AdrianSosic, I have updated both benchmarks to match the coding convention with other scripts in the repository.
There is one thing that I couldn't quite figure out with, benchmark for transfer learning. In transfer learning, I want to work with different initial data sizes. But, initial_data argument is only used in simulate_scenarios, is there a way to do this elegantly? (Line 202-216 in Hardness benchmark)
Hi @AdrianSosic, I have updated both benchmarks to match the coding convention with other scripts in the repository.
There is one thing that I couldn't quite figure out with, benchmark for transfer learning. In transfer learning, I want to work with different initial data sizes. But,
initial_dataargument is only used insimulate_scenarios, is there a way to do this elegantly? (Line 202-216 in Hardness benchmark)
Hi @ritalyu17, thanks for pinging me. Yes, there is an easy way to handle the initial_data problem: instead of passing a fixed dataset, just pass an iterable of datasets and omit the n_mc_iterations argument. That way, one MC run will be performed for each dataset you pass.
Once you've included the change, ping me again and I'll have a look at the code 👍🏼
Hi @ritalyu17, let me know when changes are incorporated and the branch is rebased 👍🏼
Hi @AdrianSosic, thanks for the suggestions. This pull request is ready for review.
@AVHopp we need an assessment whether this code is even integratable and an assessment of what still needs to be done OR simply abandon it in the absence of OP
Thanks for following up on this. The code was ready for review in March, and I’ve been keeping it on hold as I understood priorities were elsewhere. Happy to pick this back up and adjust as needed.