FRP application

[mlvl36667]

Researching MEV-boost auctions with machine learning...

[sarahalle]

Thanks for submitting this proposal! Our team is reviewing it and we will get back to you with questions or feedback

[ababino]

Thank you for submitting the proposal. Please consider the following feedback.

Your choice of SVMs as the main classification algorithm for your project is not well justified. You acknowledge that you do not know the degree of accuracy that SVMs can achieve in this setting, and you do not provide any evidence or references that support their suitability or performance. We recommend that you either provide a clear rationale and empirical validation for using SVMs, or explore other alternatives that may be more appropriate or effective for your problem domain. For example, you could look into existing approaches that use reinforcement learning, or deep neural networks to model and optimize bidding strategies. The following paper might be a good start: https://arxiv.org/abs/2106.03215

• Your assumption that no rational bot bids more than the maximum expected MEV of the given block is questionable. You do not account for the possibility that some bots may bid higher than the MEV due to subsidization, active search, or other factors that may influence their behavior. You also do not consider the dynamics and interactions among the bots, and how they may affect the bidding strategies and outcomes. We advise that you revise your assumption and incorporate more realistic and complex scenarios that reflect the actual behavior and incentives of the bots. A simulation framework was recently released by colleagues at the Ethereum Foundation that might help you gather more insights and even generate synthetic training data: https://github.com/M1kuW1ll/MMASim
• Overall, based on the above comments, we recommend you allocate more time researching existing approaches and building a few models you can compare.

We hope that you find our feedback useful and constructive. We encourage you to revise your proposal and address the issues that we have raised. We look forward to seeing your improved version and hearing more about your project.

Sincerely, Jonathan, Andres, Danning, and Elaine

[sarahalle]

Hi @mlvl36667! Do you plan to revise the proposal based on the feedback from Andres and ask us to review the updated proposal, or should I close this PR?

[mlvl36667]

I am working on the revision. Can I get back to you in a few days? I have some new ideas, I accept that SVMs might not be the best option.

[mlvl36667]

I updated the proposal. Sorry for taking this long...

[sarahalle]

Thanks for submitting the updated proposal! We are reviewing it and will get back to you with any questions here.

[jopasserat]

Thanks for reshaping the proposal @mlvl36667 ! Looks great! A few clarification points below if you don't mind:

• can you please clarify what you mean by "compare the characteristics of the two models" -> DSIC vs revenue-maximizing auction models?
• "It is known that as the number of participants increases" -> ref :thinking: ?
• You seem to have now opted to use regretnet as the main ML architecture for your project. Can you please include in the plan a literature review on regretnet vs preferencenet and its other extensions (as listed in the intro to the preferencenet paper)? This might even warrant comparing a few of these architectures in case multiple ones happen to be suitable to your problem.
• "unlike prior works using synthetic bids" -> are you referring to the EF's simulation model? a fair comparison would be most welcome as part of your evaluation indeed :)

[sui414]

FYI sharing this: eden network opened access to all the bids historical data - feel free to reach out to them if you need it for the FRP

https://x.com/EdenNetwork/status/1783128536737333266

[mlvl36667]

• "compare the characteristics of the two models", Let me explain this a little bit more. So I guess this: https://github.com/saisrivatsan/deep-opt-auctions/blob/466e154b123921c142ee811216d0e64415bf31a5/regretNet/trainer/trainer.py#L352C19-L352C20 runs the test data set through the neural network and this: https://proceedings.mlr.press/v97/duetting19a.html tells us that the auctioneer knows the distributions of the valuation functions, which is not easy to quantify for PBS builders Bids are the only observable quantities. My idea is to train RegretNet with different real distributions of valuations (e.g. how much a builder values a certain block), throw the bids (or at least a subset of them) through the networks and compare the allocation/payment rules to the winning bid. The answers I would get would look like: "Assuming that the builders have ... valuation functions, this block should have been allocated to ... builder for a payment of .... according to RegretNet". This model can be easily extended to the case of more than 1 (consecutive) slots, as RegretNet can calculate N bidders/M items scenarios.

• https://arxiv.org/abs/2202.13110 Table 1 and Table 11

• I will revise the proposal

• Yes, I will include this aspect in the modified proposal.

[jopasserat]

If you can finely describe what data points you'd like to collect, we might be able to point you to existing data vaults that could save you from doing most of the collection yourself.

[ababino]

Hi @mlvl36667,

Thanks for the update and the comments!

I still do not understand what you mean by "characteristic of the two models." What would those two models be? Are you referring to a trained RegretNet? or something else? Could you be specific about that? Also, what do you mean by "characteristic"? Could you provide an example?

[mlvl36667]

@ababino I will try to explain it more clearly. I thought about the model a little bit more yesterday. So I guess RegretNet is a deep neural network that can give you the allocation rules (who gets which item, in our case there is only 1 item) and the payment rules (how much should one pay for the item) for a set of bids (and assumed valuation distribution) and does this while keeping the regret metric minimal (participants don't pay more than their true valuation and they get the item if they submit their bids according to their true valuation) and the revenue metric maximal. I can do the same thing by running the PBS relay bids through the network and checking whether the allocation/payment rules are aligned with what the relay dictates. This is what I meant when I wrote "comparing the characteristics". RegretNet has been tested using random bids and this could be done using the actual bids from PBS relays. I expect that the current allocation/payment rules on PBS relays are not aligned with the incentives that were uncovered with RegretNet. Does this make sense?

[mlvl36667]

@jopasserat I updated the proposal.

[sarahalle]

@mlvl36667 Given questions about the proposed methodology, we are not able to fund this project. Specifically, usefulness of RegretNet was flagged as questionable given that the MEV-Boost auction is a single-item auction. Thank you for this submission and for working with our team to explore iterations!