API for getting more debugging info

[avh4]

When tests fail, it can be useful to get more information about what led up to the failure. What should such an API look like? What information should just always be shown in all errors, and which should be opted-in to?

For opt-in debugging features, the API could maybe be something like withDebugging [ ProgramTest.Debug.MsgLog, ProgramTest.Debug.PendingHttpRequests] (withDebugging : List DebugOption -> ProgramTest model msg effect -> ProgramTest model msg effect)

Possible information to show (please comment below about which of this actually seems useful for tests you've had to debug):

• history of all Msgs that were processed
• history of all effects that were simulated
• details about the browser back and forward history
• full rendering of the current view
• full rendering of the view (possibly filtered by some Query.find at each step in the test)
• pretty print of the program's Model at each step in the test
• history of steps performed in the test (the ProgramTest API calls)
• details about all pending HTTP requests
• details about all past (possibly already resolved) HTTP requests
• details about all pending scheduled Tasks
• details about all past scheduled Tasks
• details about all unchecked outgoing port values
• history of all simulated incoming port values (I guess this would already be included in "history of steps performed in the test")
• history of all previously checked outgoing port values

TODO

• [ ] refactor internals to have a data type that represents the test log
• [ ] record information in the log
  • [ ] identify the different pieces of information to record
  • (TBD) record each piece of information
• [ ] display information
  • [ ] identify the different subsets of information to display
  • (TBD) implement display of each piece

[avh4]

An existing helper from coreyhaines

{-| Helper function to show history-like information in the test output.
It shows:
    Incoming Message, Incoming Model
    Outgoing Model
    Outgoing Effect
Usage:
    addDebugToUpdate Main.update
-}
addDebugToUpdate : (Msg -> Model -> ( Model, Effect Msg )) -> (Msg -> Model -> ( Model, Effect Msg ))
addDebugToUpdate update =
    \msg model ->
        update (Debug.log "Incoming Message" msg) (Debug.log "Incoming Model" model)
            |> Tuple.mapFirst (Debug.log "Outgoing Model")
            |> Tuple.mapSecond (Debug.log "Outgoing Effect")

[adamnfish]

I'd love to be able to export a Debugger log that I can import into an Elm App running in debug mode in the browser. I'd observe that this would automatically provide most of the above, and more. It would also do so without requiring any new tooling, or any new API surface area for this library beyond the ability to do the export. This would mean the complexity for providing these features to developers can live in the existing Elm tooling ecosystem as well as allowing developers to use the same techniques debugging these tests as they would when debugging the program itself.

There are use cases that this approach would not address, but I think that even here there may be benefits to this approach. If the Elm-debugger import/export format becomes a standard for this kind of program-flow introspection then features added on top of that file format in service of elm-program-test's requirements might be just as helpful for people working outside of this library.

On top of all of this, elm-program-test also becomes a way of indirectly driving a real browser. Today developers must manually exercise a program's flows in-browser and export the debug log from there if they'd like to iterate on the design of a program's various states (for example). If elm-program-flow is encouraging developers to automate this work then it makes sense to see the benefit on both sides of the command line, as it were.

My apologies if this is already possible, and thanks very much for creating this tool.

[allanderek]

Most of those suggested would be useful in some scenario, but I think the list of Msgs and the Model states (essentially what you get in the Elm debugger) would be the most generally useful. Particularly if you've written a failing test first, and so you're trying to debug the program being tested rather than the tests themselves.

[avh4]

I had an idea here that the program test state should maintain a log of all the possible debugging information that might be wanted, and then there will be a set of flags that will be used by a rendering function that will select which subset of the full information will be displayed.

Then assertion functions will be able to automatically turn on certain display flags (for example, expectHttpRequest will turn on the "show HTTP requests" flag), and then there can also be a set of functions (potentially in a ProgramTest.Debug module) that users can use to explicitly enable the flags at any step in the test to add additional information to be displayed in any subsequent failures.

[avh4]

Here's a draft of what the full log could look like for the GrammarCheckingExampleTest, with the idea being that a subset of this information would be shown, depending on which options are set.

start
  Flags = ()
  Model = { text = "", results = [] }
  effects: Cmd.none
  subscriptions:
    - port grammarCheckResults
fillIn "Enter text to check"
  DOM event: /body/input#main onInput
  Msg: ChangeText "The youngest man the boat."
  Model = { text = "The youngest man the boat.", results = [] }
  effects: Cmd.none
  subscriptions: 
    - port grammarCheckResults
clickButton "Check"
  DOM event: /body/button onClick
  Msg: CheckGrammar
  Model = { text = "The youngest man the boat.", results = [] }
  effects:
    - port checkGrammar "The youngest man the boat."
  subscriptions:
    - port grammarCheckResults
ensureOutgoingPortValues "checkGrammar"
  ✓ "The youngest man the boat."
simulateIncomingPort "grammarCheckResults"
  subscriptions:
    ✓ port grammarCheckResults
  Msg: GrammarCheckResults [ "Garden-path sentences can confuse the reader." ]
  Model = { text = "The youngest man the boat.", results = [ "Garden-path sentences can confuse the reader." ] }
  effects: Cmd.none
  subscriptions:
    - port grammarCheckResults
expectViewHas
  ✓ text "Garden-path sentences can confuse the reader."