Feedback Ideas

[Thortok]

I'm new to FF and by extension new to crafting, but I've found this site and love it in theory.

To preface, I started a character on my girlfriend's account and got to at least level 15 in all crafting classes. Now that I have my own account and a better understanding of the game (you mean quality increases xp even if it doesn't make an HQ item? Awesome!) I've been happily using this site to try to optimize my every crafting! But I've run into a couple snags and I've had a few ideas, so let's get started! As a programmer myself, I would even be happy to volunteer to help make these happen! I want to benefit from optimized crafting just as much as the next guy. =P

1 - I have no idea what algorithms you're using or how this works, but I'm not certain why so many convoluted calculations are necessary. You have a list of starting conditions, you have a set amount of 'steps' you can take based off those starting conditions, you can assumedly calculate the outcome of each individual step. Based off that, you can then calculate the outcome of each possible combination of steps.

Ergo, you simply arrange the steps in every possible way, prune some that don't make sense, sort the results, and pick that which rises to the top as the best outcome with the highest chance of success. Then you have your series of steps. This 'random guessing/solving' thing where a potentially different outcome every time I click the solve button is confusing.

Would it be possible to explain why you don't do it this simpler way? Why the random method instead of the 'try everything and see what works best' method?

2 - Inaccuracy of data. Most of the time it works great, but occasionally I'll have either a recipe that finishes too quickly, with plenty of cp and durability left to use to increase quality that was all wasted, or a recipe that the solver says is a 100% chance of success and I'm like 2-3 progress short and I lose my mats.

I could simply report when this happens so you can fix it, but I've seen inaccuracy reported twice here with as much stats as you could ask for to help correct and the response is that being off by a couple points is acceptable.

I firmly disagree. You say you get all this by reverse engineering, and here's someone supplying you more data to reverse engineer with, and you just ignore it? Why? I assume it's because the system is based on an reverse engineered assumption of what your results will be given the stats, and here or there it's off by a little since it is just a guess after all.

Well, my idea for a solution goes like this. Allow a system called 'spot correcting.' If all your stats are entered in correctly, but the in-game results don't reflect what the optimizer expects, allow an override called 'spot correct.' It will only work for YOU, and only work if ALL of your starting stats stay the same, but it will then use your spot-corrected values for progress and such instead of the guessed ones, when coming up with a crafting solution.

While this would be individually done so as not to allow corruption of other people's experiences, each spot correct could (and should) be tracked, to allow you to collect data about what people are spot correcting and why. The more data you have, the easier it is to reverse engineer. Either for a better 'guess' overall for the crafting formula, or simply applying the most-commonly-used spot corrections into the default code so users no longer have to make them individual overrides and other users get to use the better data automatically.

I don't see a reason NOT to do this. You allow users to individually correct their own situation immediately and with no admin intervention necessary, while also collecting data for addressing the entire system as a whole. Why wouldn't you?

It would be just as easy for the user to report the data's inaccuracy via a spot-correct as it would be to adjust their craftsmanship or whatever to gimmick better results. It would also achieve a much better accuracy for the individual user immediately instead of having to 'guess' at what to adjust their craftsmanship to in order to get best results. Again, why NOT do this?

3 - Prune bad results. I don't think using steady hand 5 times in a row really does anything except waste time and cp. Master's Mend is better used to restore 30 durability with, not just 20. I can give other examples, but really, the system should be smart enough to prune these out. Especially if it starts from the 'what works best' algorithm to begin with.

In fact, all you would really have to do, is to take the 'every possible method' algorithm, and then when two methods have the exact same results for quality and success rate and in cross-class abilities chosen to be used, then only display the one with the least amount of steps to complete. That will automatically prune stuff like the 5 steady hands in a row.

4 - Maybe I don't understand how, but I'd like the option to force 100% success rate. Yes, I like the idea of increasing my quality as much as possible to get the extra xp and the chance of a HQ item, but I don't want to waste my mats, some of which may have been rather annoying and time consuming to get in the first place. Especially if the mats were HQ as well.

I know other crafters might want to risk success by trading for an increase in quality. Let them. Make it an option. Options please everyone. In fact, if you use the 'what works best' algorithm, you can display a list of different solutions instead of just one. Sort the list by user's preference, quality instead of success rate, or vice versa. Or a checkbox filter for 'only 100% success when possible.'

In some cases, a solution with a low chance of success leaves me some durability to manually finish up with. In this case, the solution's chance of success could increase enormously simply by adding 'extra' synthesis steps at the end. Since this also has a chance of extra steps occurring after the item is already finished, warn the user that they will have a chance for these 'extra' steps when they select the 'max success rate' option.

5 - I don't understand the purpose of selecting your class twice. Once for your character stats and once for your recipe. Can we just assume that a carpenter is only ever going to craft carpenter recipes, etc? I've run into a few bugs and issues before realizing I forgot to click the second time.

Also, changing class should erase the current solution. I've run into bugs switching to a class that's lower level and doesn't have an ability in the current solution. I've had to edit the sequence and clear it manually. That should be automatic on class swap.

Another user-friendly feature would be to wipe the recipe stats on class swap, or better yet, switch to the recipe stats for the last-used recipe for that class. As a quality of life improvement, making the name of the current recipe a bit bolder and stand out a bit more would be helpful, too.

6 - Option to disable the sound effect in the macro by default. The echo is great, but the sound effect gets old after hearing it so many times.

That's all my main ideas for the site's current functionality, that I can think of right now anyway. But I did have an idea for an overhaul of the purpose of the site to begin with.

You currently have an issue with 'Tricks of the Trade' and other conditionally-based abilities, yes? The ability doesn't work well in a macro because it can only be used at certain times and such.

The idea I had was less of a macro maker and more of a step-by-step recommender of what step to take next. The way it would work is this. You start off the same, plugging in character and recipe stats. Then you hit synthesize both on the site and in-game. Your abilities are listed on the site, with one highlighted. In-game, you hit the recommended ability. On the site, you hit the ability you used (hopefully the recommended one, as that's the purpose after all). If the ability you selected has 100% success rate, it moves to the next step. If there's a chance of failure, the user reports whether it succeeded or failed. The optimizer then displays current progress (which should match the game exactly, allowing for a spot-correct if it doesn't!) and then recalculates based on current progress and remaining cp/durability/etc, and recommends the next step. Repeat until item is crafted.

The potential for on-the-fly correction in case of stuff like a failed synthesis is enormous. It also allows for things like tricks of the trade to work. The user can report the current condition and the site can use that information to optimize the next step, whether that should be a tricks or something else, based on all the information it has access to.

Obviously crafting would go a lot slower, but this would maximize the quality and success rate the way nothing else can. It would be the best way to attempt difficult recipes. The macro maker would still be an option for simpler stuff, of course.

Also, this entire 'optimize on the fly' method would work exceptionally well with the earlier spot-correcting idea. If the site's progress doesn't match the in-game's progress, it can be corrected even before the item is finished being crafted! The rest of the synthesis can be optimized correctly and easily, just from a simple spot-correct! And again, the data you collect would be quite helpful for reverse engineering, particularly on when an item's condition changes, which currently there's not much data on, seems like.

You can even record data on actual success rate versus what the game claims the success rate is, how often and what potentially causes a change in condition, etc. All this metadata could be used to gain greater understanding of the entire crafting system as a whole, which could easily be looped back into probabilities and statistics the optimizer uses to recommend in the future. Reverse engineers love data. =P

You can even set the optimizer to dynamically make decisions based off that data. 'Most users who get X progress per cast on Y recipe wound up using Z sequence for best result.'' And you can use that data to recommend cross-class abilities to equip before the craft even begins.

Just be careful that the majority of your data is authentic and ignore outliers and griefers and such. But at that point the system reaches the level of magic and learns more the more it's used. At that point the system is doing the reverse engineering FOR you. And wouldn't that be great?

Again, I'm willing to put my money where my mouth is and actually help code these suggestions if you like. I'm not sure how proprietary you feel about this project.

[Thortok]

Just found your beta version, so that takes care of point 6, and point 5 can be changed. I'd like to be able to see/adjust the recipe on the solver page in addition to the simulator as I generally only use the solver and not the simulator. I can probably get used to it how it is but it'd still be nice to skip that step. =P

I also think the visibility of the currently selected recipe could be improved, and that the recipe name and stats (and heck, character stats) should be listed in the logs. That'd make reporting errors for spot-correcting easier, if you don't add in the functionality of spot correcting yourself like I suggest. =D

[Thortok]

Also, after messing around with the beta, points 1 and 3 don't seem to be issues, although I'm still concerned with point 4. So for the record that's points 2, 4, and an edit to 5 remaining. =P

[doxxx]

Thank you for your feedback.

Regarding inaccuracies, it is not as simple as you may think. The interactions between the different variables are a complex black box. @lokyst uses various numerical methods to determine how these interactions work but it will never be truly accurate simply because we lack access to the game's source code or some other authoritative description of the inner workings. One of the most pernicious problems is that the game rounds/floors results at various stages of the calculations, which makes it very hard to determine the factors by linear regression. These and other complications make it difficult to model the system with complete accuracy. The number of combinations of variables makes your spot check idea infeasible in terms of memory usage, and the resulting data would not be particularly re-usable due to the result only occurring under a very specific combination of variables. If you find that the results are off by one, you can try using the temporary stat bonuses (on the right side of the simulator status) to tweak the results.

Regarding 100% success rate, in the options there is a setting called Reliability. This should be 100% by default and is used by the solver to determine the fitness of a sequence. If the solver is not producing a sequence with the desired reliability, then it may not be possible to achieve for that recipe at your level and stats. If you have any specific examples where you believe it should be able to achieve the desired reliability and is not, please file an issue with the specifics.

I like your idea of remembering the recipe last selected for each class and I can also add an option to disable the sound effect in macros.

The step-by-step recommender idea is something that we've been thinking about for a while. We call it the Magic 8 Ball. A couple of the features you describe would require backend server support which we don't currently have -- the site is entirely static and executes all simulations locally in your browser. But it's certainly not impossible. However, it is only an idea at this time and our top priority is improving the model and the usability of the current features of the site.

[Thortok]

Thanks for the reply!

Even if the spot correct isn't reusable due to being too specific a scenario to pull data from, I still think a client-side override option would work better than gimmicking the system by tweaking stat bonuses you don't actually have some unspecified amount until you get the result you want. I'm not getting the memory issue, as it would be using static values instead of the numerical methods you mentioned.

Perhaps you're saying it would remember a spot correct for future visits, which it doesn't really need to. The reason I call it a 'spot' correct is especially while leveling crafting where most of the inaccuracies are likely to be, you tend to not have the exact same stats for very long anyway.

Essentially it would be something in the options menu for the class in question, and you click to enable and say 'progress of an unbuffed basic synthesis = X' and 'progress of an unbuffed basic touch = Y' and if you change any stats of the class you set the spot correct on, it's forgotten forever and you'd have to adjust the spot correct again, if necessary.

The idea of remembering these variables doesn't seem like that much data-per-spot correct: class level, craftsmanship, control, recipe level, recipe difficulty, and amount of progress or quality per step, shouldn't be too hard to store in a spreadsheet. I'm not sure recipe difficulty is even needed. If progress while buffed is inaccurate even after unbuffed has been corrected, there may be cause to have a variable listing the buffs enabled at the time of the ability's use. But that would be it.

As you say, coding in a 'data recorder' into the pure client-side app to be stored on a server somewhere would be harder than the actual storing or perusing of the data. I didn't realize it was pure client-side. The server recorded aspect can totally wait on the back burner, but I still think a client-side correction method would be much more direct and accurate instead of 'tweaking' stats to hopefully get the result you want through trial and error. =P

I'll have to check the reliability thing. I wasn't sure what that setting was for (could use just a bit more explanation in your instructions for stuff like that) but I didn't notice this quite so much on the beta site. On the non-beta, I've seen it 'solve' a sequence with 100% success down into something with like 40%.

Again, my crafters are low level so a lot of these issues may not occur at higher levels. Still, it should be working from the ground up, and the same bug that occurs at low levels may happen at higher levels and just be harder to notice because of the additional complexity.

I'm glad the 8-ball idea is on the table even if it's the back burner. Again, I'd be happy to help if/when I could. I'm new to github, but I'm assuming that once I know how to use it correctly I can dive right in and start helping?

[Thortok]

Also, would it be possible to be a bit clearer on what some of these options do?

• Montecario runs - I know increasing this number gives a higher potential degree of accuracy in the log that occurs after a solution is generated, but does it have any effect on the generating of the solution at all to edit this number?
• Use Conditions/Override on Condition = ?
• Max Tricks - Referring to the tricks of the trade ability that can't be macroed very well, how many times to use it in a sequence. I currently have set at 0.
• Reliability 100% - You said earlier this is the desired success of the actual item. Clarifying when reliability refers to the item being crafted at all versus HQ reliability could use some work.
• Algorithm - Choosing which algorithm to use. Gonna give complex a try.
• Population = ?
• Generations = I know raising this number makes the solver work longer before revealing a result so I assume it's how many times it tries to find a better solution than what it's currently figured out. Is there any way we can dynamically set this? As in, if the 'best' solution is the same thing X amount of times in a row, just go ahead and stop?

The macro options are obvious and the debugging I assume is for debugging and not intended for normal client use so I won't worry about those. But an explanation of the others would be helpful. =D

Next is this part of the window:

First off, things aren't lined up correctly. The 50/48 thing doesn't match the same line as 'Progress' for instance, it's lined up with the bar. That same problem goes all the way down.

The third bar confused me. For awhile I thought it was chance for HQ, then I realized it represents the CP (the misalignment of the numbers on the right contributed to the confusion.) The bar to represent CP feels kind of unnecessary as it doesn't show immediately under the other two bars in the game. On the one hand an idea of how much CP is going to get used is useful information but on the other it just feels confusing to have it as the exact same kind of visually represented bar as the other two. Main reason? You want the other two bars to be as full as possible. But this one you like it better when it's empty. Feels weird overall.

Underneath is HQ, X% reliability (which again isn't clarifying that it's the HQ reliability, I eventually figured that out though) and then 100%. Not sure why the 100% is sitting there, it seems to always be there. Something that's always there isn't really useful. Unless it's saying the far right of the bar is 100%, which is kind of self-explanatory and doesn't need to be pointed out.

The main things I would change about this section:

Align Progress and Quality so the values line up with their labels and not with the bars. Drop CP and HQ% completely or redesign to show them somewhere else than this window. Remove the CP bar unless you redesign it to be visually different from the progress and quality bars. (Perhaps shorter and purple like it appears in-game.) For HQ% of success, somehow tying that in to the quality bar would work a lot better (again similar to in-game.) Right now it's under the CP bar which is just weird. The percentage goes up as the CP bar goes down, again weird.

Next is this part:

Not sure why this is even here. Never changes. Doesn't do anything. Seems to waste space. Possibly something that may eventually go with your 8-ball thing, but right now it's just blah.

Finally:

I often find myself on the solve page saying "Wait, what did I just find a solution for?" I want to double-check that I am actually making X recipe before I copy the macro and use it. Currently this requires switching back to the simulator page. Simply displaying the information on which class and what recipe would be great, although you could also throw in starting quality.

I also feel that the rest of the line could be opened up to having a flatter, more horizontal list of the currently selected crafter's stats, and then you could make the recipe name a drop down like on the simulator page, and then all functions could be done from this one page instead of going back and forth.

I get that the idea is to 'start with' a simulation and then move to the solver to 'improve' it but honestly I just hit solve with a blank simulation and let it figure out something I can use. Perhaps this isn't as feasible at higher levels with more abilities, but at low levels it seems to work just fine. Being able to skip the 'simulator' process would work very well for me. After all, the optimizer's supposed to be smarter than anything I can come up with, right? =P

Thanks again for taking the time to read this feedback!

[doxxx]

[woops wrong account]

Don't have a lot of time to comment right now, but a few quick points:

• The options do need descriptions.
• Regarding the layout of the status pane, it's meant to mimic the layout of the in-game crafting window. The progress/etc. numbers lining up with the bars is purposeful, but I can see how that could be confusing. Unfortunately, it's very difficult to get a precise layout that also adapts to different browser window / device sizes. I'll see what I can do about improving it.
• Looking at the HQ/Reliability line, I can see why you're confused. It's actually saying HQ: 15%, Reliability: 100%. The spacing is again trying to mimic the crafting window but it's stretched too far because I'm trying to make it adapt to the window size. This certainly does need improvement.
• Durability is more useful in the simulator when editing the sequence (try it!), Condition is kinda not useful right now -- part of that magic 8 ball idea -- so maybe I should just hide it.
• Good point about the recipe name in the Solver page.
• I am not quite happy with the back-and-forth between the Simulator and Solver page.