cms-dev
See output of testcases for a submission in AWS
Is there a way to see what the stdout output of each testcase was for a submission? Right now, it only shows memory, execution time, and a few other details. However, it does not say what the output was. There is a Standard Output section, but it is always blank.
Knowing the output of the submission is helpful for debugging (suppose one of the test cases is wrong).
Thanks, -Amar
I would love if AWS allowed to do this.
A very good solution would be a button that, when pressed, makes AWS re-execute the submission on that specific testcase and then downloads the output.txt
I can implement this. However I'm wondering if your solution would be better than just storing the output of each testcase in Postgres the first time.
On Apr 16, 2017, at 5:18 PM, William Di Luigi [email protected] wrote:
I would love if AWS allowed to do this.
A very good solution would be a button that, when pressed, makes AWS re-execute the submission on that specific testcase and then downloads the output.txt
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
Storing the output for every submission for every testcase is unfeasible, it would require too much space. A proposed workaround is to store part of it, e.g. the first 200 bytes of outputs, but it's not very good imho. I would give the option to recompute and download the entire output anyway, even if the 200 byte solution is implemented
Il lun 17 apr 2017, 03:58 Amar Ramachandran [email protected] ha scritto:
I can implement this. However I'm wondering if your solution would be better than just storing the output of each testcase in Postgres the first time.
On Apr 16, 2017, at 5:18 PM, William Di Luigi [email protected] wrote:
I would love if AWS allowed to do this.
A very good solution would be a button that, when pressed, makes AWS re-execute the submission on that specific testcase and then downloads the output.txt
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/cms-dev/cms/issues/755#issuecomment-294389354, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOc8eZKkaVDkF7XHPeftgNzQZBhpUKqks5rwscqgaJpZM4M-NTq .
I wouldn't, once you download it there is not much point in keeping it... but I coukd be convinced to store it if a reasonable use case comes up
Il lun 17 apr 2017, 06:37 Amar Ramachandran [email protected] ha scritto:
@wil93 https://github.com/wil93 Would you save the output in the db after the re-execution then?
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/cms-dev/cms/issues/755#issuecomment-294403540, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOc8XrDULMooIx5fy1nlTcvVJaSlV3Lks5rwuyYgaJpZM4M-NTq .
This is a duplicate of #109. Since discussion has been going on here, I'm closing that one.
My two cents. I see a couple of ways to implement this without being too obtrusive:
- add a functionality to AWS that, given a submission and a testcase, builds an usertest with their files, assigns it to a fake system user and sends it to ES (upside: no changes in ES, Worker and TaskTypes needed; downside: need a fake system user; source code is compiled again)
- add a functionality to ES to send a custom Job for evaluation (built from the executable of the submission and the data of the testcase, with the keep_output flag set to true), which is inserted in the queue with very high priority.
I prefer the second solution, even though it's more work and it makes ES even more messy. Another side note: the output file has to be stored in the database as there's currently no other way for the workers to send back data. (I oppose storing the content of a file inside the Job object).
I also prefer the second solution, and I believe it could be a good excuse to start refactoring ES by having it use an established job queue, battle-tested and production ready, instead of reinventing it. For example: https://github.com/celery/celery
Transitioning to an external "professional" task queue and broker system has been considered on-and-off since at least 2012. We considered "raw" RabbitMQ and ZeroMQ back then, still with the idea of building our own custom task queue system on top. Celery seems to be an off-the-shelf solution for the second part, but it's still based on RabbitMQ or another broker system.
The reason that kept us from pursuing it (other than limited manpower) was that we didn't want to rely on a system that could become a single point of failure for a large part of CMS. We've always designed our services in a way that allows for any of them to die with limited impact on the other ones (and with the ability to easily respawn the dead one and have it continue from where it left off). Having a broker service mediate messages between, say, ES and the workers means that a failure in the broker server would prevent all these services from operating even though they are still alive.
It's true that we already have such a point of failure, and it's PostgreSQL. It's quite hard to do without it, though. In critical settings the danger of it failing can be mitigated by using the high-availability systems that PostgreSQL provides, but they're not particularly easy to configure and we see them as an unfortunate necessity. I imagine that RabbitMQ also has similar high-availability systems.
That said, we might still want to consider RabbitMQ (and, possibily, Celery or something like it). They might allow us to improve the performance of ES, or allow us to have multiple ESs run in parallel. I don't know. I guess the main problem remains the lack of manpower to handle a transition of a critical service. I don't see it happening for v1.4.
Also, I don't think that queue management is the slow point of ES. Amir suggested instead splitting ES in several subservices - the idea is that what prevents ES to be parallelized is queue management, but the slow parts are the rest (that deal with the db). The problematic part of this (but it would be the same with celery I guess) is to manage the different stages of a job making sure there are no duplicates, especially when external events happen (sweep, invalidates).
On 20 April 2017 at 01:13, Luca Wehrstedt [email protected] wrote:
Transitioning to an external "professional" task queue and broker system has been considered on-and-off since at least 2012. We considered "raw" RabbitMQ and ZeroMQ back then, still with the idea of building our own custom task queue system on top. Celery seems to be an off-the-shelf solution for the second part, but it's still based on RabbitMQ or another broker system.
The reason that kept us from pursuing it (other than limited manpower) was that we didn't want to rely on a system that could become a single point of failure for a large part of CMS. We've always designed our services in a way that allows for any of them to die with limited impact on the other ones (and with the ability to easily respawn the dead one and have it continue from where it left off). Having a broker service mediate messages between, say, ES and the workers means that a failure in the broker server would prevent all these services from operating even though they are still alive.
It's true that we already have such a point of failure, and it's PostgreSQL. It's quite hard to do without it, though. In critical settings the danger of it failing can be mitigated by using the high-availability systems that PostgreSQL provides, but they're not particularly easy to configure and we see them as an unfortunate necessity. I imagine that RabbitMQ also has similar high-availability systems.
That said, we might still want to consider RabbitMQ (and, possibily, Celery or something like it). They might allow us to improve the performance of ES, or allow us to have multiple ESs run in parallel. I don't know. I guess the main problem remains the lack of manpower to handle a transition of a critical service. I don't see it happening for v1.4.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cms-dev/cms/issues/755#issuecomment-295505070, or mute the thread https://github.com/notifications/unsubscribe-auth/AAdociVryzo65egMLuKlNU281-znXAM0ks5rxqMPgaJpZM4M-NTq .
