fchollet
Ambiguous rule for 070dd51e + some questions
Rule seems pretty straightforward:
- connect all dots of matching color in the same horizontal row
- connect all dots of matching color in same vertical column (overlap horizontal lines in case of intersection)
But when I saw the puzzle, instead of thinking in terms of vertical/horizontal lines, I thought the rule was:
- connect dots of matching color in order from least distance to greatest distance
Looking at the examples, this rule would be supported and produces the incorrect result:
This ambiguity could be avoided by providing an example where two lines of equal length intersect (and the vertical line overlaps the horizontal line). For example: shorten the distance between the maroon (color 9) dots in Example 2 by so that the light blue (8) and maroon (9) lines both have a length of 6.
This brings up other concerns: How is it made clear by these two examples that color is not an important factor in determining overlap? Could I come to the belief that green must always be drawn beneath other lines? What about line lengths? Can I say that all lines of length six must be overlapped when in intersection? What about interactions between two different colored lines?
Surely I could keep going with an infinite list of increasingly complex rules. So, should an agent solving the puzzle be required to ascertain the most "simple" solution, or should any rule outside of the correct solution be necessarily contradicted by the examples? Is the latter even possible? How is this simplicity defined? Is there a method by which these puzzles are checked/proven to be solvable (with only one solution)?
I am new to the arc challenge and have not read all of François Chollet's paper, so my questions are not a critique but simply an attempt at understanding. Please let me know if you have any thoughts! I'd be happy to submit a PR if other people believe it to be necessary. Thanks, Max
Hi. You pose some tough questions!
"How is it made clear by these two examples that color is not an important factor in determining overlap?" Who said color order rules aren't expected to be part of the information acquisition need of an AGI?
"So, should an agent solving the puzzle be required to ascertain the most "simple" solution, or should any rule outside of the correct solution be necessarily contradicted by the examples?" I would expect that some kind of optimization is occurring in any solution. Simplicity for your approach might be depth of rules actioned, for example.
"Is there a method by which these puzzles are checked/proven to be solvable (with only one solution)?" Yes. If your output equals the answer. That's the verification method.
"I am new to the arc challenge and have not read all of François Chollet's paper" A great place to start is to read the entire paper, a few times.
Hope this has helped. I think the issue can be closed. Cheers GM
Who said color order rules aren't expected to be part of the information acquisition need of an AGI?
Unfortunately I don't think you answered any of my questions. I understand the premise of how these models should be solved, but I was curious how to differentiate between solutions when each solution could be valid given the implied rule set. To your first response, I obviously understand color is important to the arc challenge, but I was curious for this specific example how we should know for certain that color actually wasn't important in the solution.
I don't believe the issue has been solved or should be closed.
Thanks, Max
I wonder if you could add some kind of complexity score to rules, so that if two rule sets give a solution in examples, then the chosen is the most simple? I think the creators of the test didn't assess the dataset for multiple solutions. Does this happen in any other example? (I'm not using DSL logic approach so don't know).
Originally I came to a solution similar to the issue's author: the longer line overlaps the shorter one. Following this rule leads to paint (4,10) square as red. While following the ARC rule it turned out that this square shall be grey.
I wonder if you could add some kind of complexity score to rules, so that if two rule sets give a solution in examples, then the chosen is the most simple? I think the creators of the test didn't assess the dataset for multiple solutions. Does this happen in any other example? (I'm not using DSL logic approach so don't know).
Based on the paper indeed the simplest solution shall be used "A hypothetical ARC solver... select top candidates among these programs based on a criterion such as program simplicity or program likelihood"
Also citing the paper: "we consider 3 trials to be enough to account for cases in which the task may be slightly ambiguous..." So the task's ambiguity is considered too.
Anyway, an example that separates explicitly those 2 rules would help to identify which one was used to organize the output and remove ambiguity. Having 2 lines of the same length and identifying that the vertical shall be on top of a horizontal would help.
I wonder if you could add some kind of complexity score to rules, so that if two rule sets give a solution in examples, then the chosen is the most simple? I think the creators of the test didn't assess the dataset for multiple solutions. Does this happen in any other example? (I'm not using DSL logic approach so don't know).
Based on the paper indeed the simplest solution shall be used "A hypothetical ARC solver... select top candidates among these programs based on a criterion such as program simplicity or program likelihood"
The paper doesn't refer to the simplicity of a solution to a task - but to the simplicity of the overall solution to the ARC challenge. NikolasBielski's comment was about if a solution, the correct output on a specific task, if many valid outputs are possible in respect of the given examples, must follow the simpliest rules.
I can come up with atleast four different answers/outputs to 070dd51e, that all are valid correct outputs according to the two attached examples, that I personally think applies about equally simple/complex set of rules:
- The vertical lines always overlap the horizontal lines.
- The longer lines always overlap the shorter lines.
- If "painted" simultanously from the edge cells, for each color, until they meet in the middle - the later (in time) plotted cell overlaps any already plotted cell.
- Colors decide what line overlap the other.
Somehow the ARC test only accepts applying rule set number 1 here as the correct answer - while the other three answers should be correct too? The two examples attached to 070dd51e doesn't invalidate any of the other three perspectives of the problem. In my opinion Puzzle ID: 070dd51e is flawed, atleast it should accept all different possible solutions as a correct answer.
In my opinion Puzzle ID: 070dd51e is flawed, atleast it should accept all different possible solutions as a correct answer.
The dataset was planned to have a single answer, so a pull request that proposes such an example...
that explicitly separates those 2 (or more) rules, would help identify which one was used to organize the output and remove ambiguity. Having 2 lines of the same length and identifying that the vertical shall be on top of a horizontal would help.
... will solve the issue.