java
java copied to clipboard
exercism
Build analyzer
In Exercism v3, we are making increased use of our v2 analyzers. Analyzers automatically assess student's submissions and provide mentor-style commentary. They can be used to catch common mistakes and/or do complex solution analysis that can't easily be done directly in a test suite.
Each analyzer is track-specific. When a new solution is submitted, we run the track's analyzer, which outputs a JSON file that contains the analysis results.
In v2, analyzer comments were given to a mentor to pass to a student. In v3, the analyzers will normally output directly to students, although we have added an extra key to output suggestions to mentors. If your track already has an analyzer, the only requisite change is updating the outputted copy to be student-facing.
The analyzer is an optional tool though, which means that if a track does not have an analyzer, it will still function normally.
Goal
Build an analyzer for your track according to the spec. Check this page to help you get started with building an analyzer.
It can be very useful to check how other tracks have implemented their analyzer.
If your track already has a working analyzer, please close this issue and ensure that the .status.analyzer key in the track config.json file is set to true.
Choosing between representer and analyzer
There is some overlap between the goals of the representer and the analyzer. If you want to build both, we recommend starting by building the representer for the following reasons:
- Representers are usually (far) easier to implement
- Representers can have a far bigger impact on the mentoring load than analyzers by empowering mentors
- Representers apply to all exercises, whereas analyzers usually target specific exercises or a subset
Tracking
https://github.com/exercism/v3-launch/issues/53
In this moment the Java track doesn't implement a java analyzer that satisfies the v3 requirements. This is a good starting point for a new motivated contributor/maintainer.
ATTENTION: it needs a good amount of work to reach some results
Claim this issue if you feel excited by this task. :fire:
Hello,
I am interested in working on an analyzer if it is still needed, though I would need to do a significant amount of research on it before starting.
Just to confirm, an analyzer for Java will need to be made from scratch for this? I see there is already an older java-analyzer repository.
Do you have any specific suggestions for existing analyzers for other languages that would be good to look into for preparation?
There are quite a few analyzers. The most up-to-date ones are probably the Elixir analyzer and the Javascript analyzer.
@jmrunkle and @Smarticles101 do you have any suggestions if start from scratch or upgrade the existing one ?
I believe the idea is to use the java-analyzer repo, but it may need to be re-written depending on how different the goals are here. I thought there was enough alignment that a simple re-use would be sufficient.
I can also believe that completing a java-representer (I think I saw one started) would help tremendously.
@mirkoperillo I believe the main change from v2 to v3 analyzers is the deprecation of the status of approve, disprove, or refer to mentor. Instead of returning a status, the analyzer simply returns a series of comments.
It should be fairly straightforward to port the old one to the v3 spec, it might even simplify the code a bit.
Like @jmrunkle pointed out, there is a java-representer. I am not sure what state it is in, but it would also be good to dig into for anyone working on the analyzer.
I am looking now at the representer and at a first glance I'm getting the impression it is at a good stage. I can't be sure though as I just started looking into all this.
I looked at a few doc files related to representers and analyzers but didn't come across specific features desired for representers. Perhaps if something is missing it would not be too difficult to add additional normalizers.
The representer implements the current requirements. There are a couple of minor issues opened. I think there will be more to do on it when the new mentor UI will be implemented and so it will go "officially" in production. In this moment it is a priority to have a working analyzer.
@olapokon I think that a first interesting task on analyzer could be understand what the current implementation does and what is missing against the new spec. To maintain trace of this work and leave the discussion open I think the result of this task could be documented in a new issue to permit everybody to participate and to stay aligned. What you think about ?
@mirkoperillo That sounds like a good idea. I started looking into the existing analyzer. I expect it may take at least a couple of days, but that will probably depend on the circumstances too.
I have a question regarding the automated comments that will be provided by the analyzer.
From what I understand, the analyzer is dependent on these https://github.com/exercism/website-copy/tree/47af5b309ac263629ca5c52904046f81e0cc8def/automated-comments/java so it will need to be extended every time additions are made to that repository.
There is also a directory with general comments there. Should the analyzer be able to take as input any of the existing exercises and apply the general comments and in addition the comments specific to that exercise (if they exist)? Or should it only accept exercises for which specific comments are available?
I'm not sure if this is the right place for questions of this kind.
To maintain trace of this work and leave the discussion open I think the result of this task could be documented in a new issue to permit everybody to participate and to stay aligned. What you think about ?
@mirkoperillo Should I open the issue in the java-analyzer repository? Or in java?
This issue has been automatically marked as action/stale because it has not had recent activity. Please update if there are new updates to provide.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@andrerfcsantos looks like I can take over this one. There seems to be a lot of information here for me to use to get the job done. This seems straight forward. I may or may not have some questions once I look more into the code.
@jcovington16 That's great to hear! I'd say to start looking at what's already done in https://github.com/exercism/java-analyzer
I'll have to look into it a bit more to I know the exact state of the analyzer and what's preventing it from being live. I'd start to see if analyzer has the inputs/outputs expected in the https://exercism.org/docs/building/tooling/analyzers/interface, I'm assuming that's the main issue here. While investigating the code, feel free to open issues on the analyzer repo for problems you see along the way and we go from there.
@andrerfcsantos No problem. Thanks for letting me take ownership of it. I'll get right to it.
@andrerfcsantos I see that there's an analyzer and representer built for the Java track, but not sure what state is it in. I pull both of these down and try to test the them both and go from there. I notice in the original comments that the representer was recommended to complete first before knocking out the analyzer. Would you by chance know the status of the representer? If not then no worries.
I took a quick look at both and it seems the analyzer is more developed than the representer. There seems to be some incomplete classes in the representer.
The debate of analyzer vs representer at the moment is a tricky one. It is true that working on a representer has a greater impact as the developments there will affect all exercises. However, there's currently no frontend for the representers, while the analyzer frontend already works if we activate the representer for the track. There are plans to move the frontend for representers forward in a near future, but no timeline at this time. I leave up to you which one you want to work on.
@andrerfcsantos Thank you for checking. I think to keep things scope and not scope creep. The analyzer is what needs to be built, so I'll just start there.
@andrerfcsantos @mirkoperillo The analyzer is put together, but is only set up for 2 exercises. Is it safe to say that the analyzer should handle more than just two exercises correct? I'm going to take a look at the JavaScript analyzer and see the pattern that I should follow only if it's fully put together.
@andrerfcsantos @ErikSchierboom I'm not sure if you guys saw my message above, but wanted to let you guys know that I'm still working on this and was hoping you guys could answer my question above whenever you guys have some time. Thank you
@jcovington16 Yes, the analyzer should support more than 2 exercises. Ideally we would have suggestions for all exercises, but starting with the concepts exercises would be best, I think the Javascript analyzer is one of the most complete ones, so it is good inspiration :)
@andrerfcsantos @mirkoperillo The analyzer is put together, but is only set up for 2 exercises. Is it safe to say that the analyzer should handle more than just two exercises correct? I'm going to take a look at the JavaScript analyzer and see the pattern that I should follow only if it's fully put together.
Yep, the analyzer should handle more than just two exercises.
In general, an analyzer has two types of suggestions/comments:
- Exercise-specific: these are suggestions/comments that are specific to just one exercise and you only want the analyzer to comment on them for that exercise.
- Exercise-agnostic: these are suggestions/comments that apply to to all exercises (with possibly invididual exercises opting-out from this)
@andrerfcsantos @mirkoperillo The analyzer is put together, but is only set up for 2 exercises. Is it safe to say that the analyzer should handle more than just two exercises correct? I'm going to take a look at the JavaScript analyzer and see the pattern that I should follow only if it's fully put together.
Yep, the analyzer should handle more than just two exercises.
In general, an analyzer has two types of suggestions/comments:
- Exercise-specific: these are suggestions/comments that are specific to just one exercise and you only want the analyzer to comment on them for that exercise.
- Exercise-agnostic: these are suggestions/comments that apply to to all exercises (with possibly invididual exercises opting-out from this)
@ErikSchierboom Based on your response I gathered that there should be some generic hard coded suggestions/comments that should apply to all exercises and there are specific ones that should be displayed with specific exercises. Is that safe to assume? I'm still working the issue. Been sick for a while, but I'm much better now.
Based on your response I gathered that there should be some generic hard coded suggestions/comments that should apply to all exercises and there are specific ones that should be displayed with specific exercises. Is that safe to assume? I'm still working the issue. Been sick for a while, but I'm much better now.
Well, it's fine to start out with one of the two, but yes :)
Given the current state of the analyzer and the work that has been ongoing there, I'm going to close this issue as completed.
