C2SP
Requesting a meeting to discuss plans with Wycheproof
I would like to have a meeting that discusses the plans of Wycheproof.
In its current state there is a lot of uncertainty and confusion. For example the fate of the generator code for the test vectors is unknown. As a result I'm rewriting a lot of code that would otherwise not be necessary. The format for the test vectors is unclear. Proposals I made some time ago have not been discussed so far. And it would be a good time to do so now. Communications is poor, which also leads to replication and slow down. My work in progress is currently here https://github.com/bleichenbacher-daniel/Rooterberg This effort involves a lot of duplication. It would be very useful for me to sync this and know how to avoid further duplication.
A meeting would in my opinion be a good way forward to understand what has gone wrong with the project and how to fix this.
I support this. While I understand the move of this project towards more "data-only" test-vectors, there is a lot to be lost here. While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links, like:
Bouncy Castle's ECDHC leaks private keys: Test: testModifiedPublic and testWrongOrderEcdhc in EcdhTest.
While the tests may have been outdated and testing only a few libraries they could provide great inspiration for testing other libraries that data-only test vectors do not give.
Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced". I understand the former is important for consumability of the test-vectors, without the latter you will be reinventing the wheel.
@FiloSottile @cpu
While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links
I've tried to address both points with https://github.com/C2SP/wycheproof/pull/139, thanks for flagging.
Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced".
I have zero insight into the former or current state of the generation code and don't personally see any way to make progress on that front. My perspective is that it doesn't make sense to continue to do nothing waiting for something outside of my control that may/may not happen. I'd prefer to prioritize pragmatic changes that can be done today, accepting the risk some of the work may become redundant in the future.
While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links
I've tried to address both points with #139, thanks for flagging.
Sure thing.
Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced".
I have zero insight into the former or current state of the generation code and don't personally see any way to make progress on that front. My perspective is that it doesn't make sense to continue to do nothing waiting for something outside of my control that may/may not happen. I'd prefer to prioritize pragmatic changes that can be done today, accepting the risk some of the work may become redundant in the future.
Hmm, I see, I thought there was some communication/continuity during the move from Google to C2SP. Do you have any insight on who to bug to make the code release happen?
While git history keeps the test code, it is much harder to go through it. Now, even the "Hall of Bugs" in the README contains broken links
I've tried to address both points with #139, thanks for flagging.
Sure thing.
Similarly for the test generation code. If that is not properly open-sourced it will be a pretty bad loss. For me, the priority of "linting of test-vectors" <<< "generation code is open-sourced".
I have zero insight into the former or current state of the generation code and don't personally see any way to make progress on that front. My perspective is that it doesn't make sense to continue to do nothing waiting for something outside of my control that may/may not happen. I'd prefer to prioritize pragmatic changes that can be done today, accepting the risk some of the work may become redundant in the future.
There are of course issues like https://github.com/C2SP/wycheproof/issues/106 that can be discussed without having to wait for any response by Google. The new version of the test vectors currently use the proposed format. It seems suitable to me, but maybe someone has some feedback here.
Hmm, I see, I thought there was some communication/continuity during the move from Google to C2SP. Do you have any insight on who to bug to make the code release happen?
On the contrary there was a distinct lack of communication. For example I wasn't even informed about the move for almost one year. Unfortunately, this situation continues to be the case. Regarding the generation code: I have no idea why previous attempts to get access to the code failed. I have contacted a few people at Google, who said they'll look into it.
The code that was deleted essentially consisted of two parts:
(1) code that runs the test vectors against Java providers and Javascript libraries. I'm currently extending this part to other languages for the new format. E.g., rust, more javascript libraries and python.
(2) tests that do not use test vectors. These tests can of course run independent of Wycheproof. Maybe there are some more appropriate places for them. I have a few more ideas for additional tests, but haven't had the time yet to work on them.
(2) tests that do not use test vectors. These tests can of course run independent of Wycheproof. Maybe there are some more appropriate places for them. I have a few more ideas for additional tests, but haven't had the time yet to work on them.
Maybe the Paranoid lib, as we touched upon in previous similar discussions in this repo?
(2) tests that do not use test vectors. These tests can of course run independent of Wycheproof. Maybe there are some more appropriate places for them. I have a few more ideas for additional tests, but haven't had the time yet to work on them.
Maybe the Paranoid lib, as we touched upon in previous similar discussions in this repo?
I'm not sure yet. My main focus is still on extending the test vectors and running them against a number of libraries. One issue is often to determine all the algorithm parameters and formats used by a specific implementation.
Wycheproof is a better fit for most of the additional tests. Paranoid works best if one has access to large collections of public keys and signatures and of course the necessary computational power to run the tests. I don't have that at the moment. I'm still trying to get in touch with Google with the goal to get a more stable and productive environment so that the tests can be revived. But I'm also open to other options.
To me, something like CryptoFuzz is a better fit. Originally by Guido Vranken who has taken it private. Now it seems Mozilla Security team is keeping it alive.
The advantage is that there are unified bindings to exercise a large amount of primitives in a large amount of implementations.
We have that for ECC only in ECTester.
Thanks for pointing out CryptoFuzz. That looks indeed quite nice. Unified interfaces would be great. I would already be happy if there were some common interfaces per language. Java looks good in this regard, rust has some widely used traits for some primitives but not for others. Javascript is a complete mess and this is unfortunately also the language where additional tests have the highest changes of finding vulnerabilities. Currently, I'm writing code for timing and key generation. In this case, I can just generate some JSON files containing data that can be analyzed later.
A meeting occurred between Filippo and Daniel and the direction of the project clarified.
Please, reopen. None of the issues mentioned above have been clarified. In particular it is unclear what happens with the test code that has been deleted. This code is an essential part of Wycheproof. It implements the main goal of the project. Running the test vectors against multiple libraries gives immediate results and helps a lot to debug both test vectors and tested libraries. Timing tests, which have been deleted, are a significant source for vulnerabilities in current libraries. Hence it would be a good idea to resurrect this part too.
Hello @bleichenbacher-daniel, on September 30 we had a multi-hour call in which I explained that Wycheproof (originally a Google project) was transferred by Google to C2SP with a mandate to make it a community project. That is what we are doing in #182.
We tried to find a way to enable you to contribute to the project, including offering to reserve a namespace of test vectors for you to control and to build tooling to simplify merging them with the rest. Unfortunately, according to my notes, you requested ownership of the project, routing new vector requests exclusively through you, and generating them all from your code. We could not find a compromise consistent with the goal of making Wycheproof a community project.
We decided and motivated removing the Java and Javascript harnesses and tests in #105. There's no doubt harnesses and timing tests have value! They are just not what we'd like to focus on, based on discussing and observing how Wycheproof has been adopted by downstream libraries over the years. All the removed code is available in the git history, and licensed under the Apache license, so you are welcome to fork it or integrate it into your own project. (We also went out of our way to preserve the original generator code that Google released in July, so you can use it if you want. It is also under the Apache license.)
I'd like to thank you for your many contributions to cryptography over the years, and I wish you the best with your projects. My decisions on the matter are final, but if you wish to appeal them you are welcome to reach out to the stewards. Please refrain from making off-topic comments in unrelated discussions to redirect people to your personal projects.
