makeLenses (w/ TemplateHaskell) changes the behavior of alex

[Ptival]

Hey, this is kind of a specific quesiton, but I am building a lexer using alex, and the following thing happens:

I'm using :

%wrapper "monadUserState"

And I'd like my state to use lenses:

data AlexUserState = AlexUserState
  { _commentDepth :: Int
  , _stringBuffer :: String
  }
  deriving (Show)

But, the moment I add the line:

makeLenses ''AlexUserState

I get a bunch of errors:

/home/ptival/language-ocaml/templates/wrappers.hs:243:5: error:
    Not in scope: data constructor `AlexEOF'
    Perhaps you meant variable `alexEOF' (line 263)
            
/home/ptival/language-ocaml/templates/wrappers.hs:244:5: error:
    Not in scope: data constructor `AlexError'
    Perhaps you meant variable `alexError' (line 223)
            
/home/ptival/language-ocaml/templates/wrappers.hs:245:5: error:
    Not in scope: data constructor `AlexSkip'
            
/home/ptival/language-ocaml/templates/wrappers.hs:248:5: error:
    Not in scope: data constructor `AlexToken'
            
/home/ptival/language-ocaml/templates/wrappers.hs:495:14: error:
    Not in scope: type constructor or class `AlexAddr'
            
/home/ptival/language-ocaml/templates/wrappers.hs:499:15: error:
    Not in scope: type constructor or class `AlexAddr'
            
/home/ptival/language-ocaml/templates/wrappers.hs:503:15: error:
    Not in scope: type constructor or class `AlexAddr'
            
/home/ptival/language-ocaml/templates/wrappers.hs:507:15: error:
    Not in scope: type constructor or class `AlexAddr'

Is there some reason why this cannot work?

Right now, it is not a huge deal, because my AlexUserState does not need any alex-defined data type, but if I wanted to have some AlexInput in my state, for instance, it seems that this could be desirable.

I'm sorry if this is addressed in the documentation, but I did not see such a thing. It's also not a huge deal as it is not preventing me from working, I'm more worried that this is an unwanted behavior.

Best regards!

[simonmar]

Not sure, it could be some bad interaction between the generated code and TemplateHaskell.

[Ptival]

Oh, it does turn out to be annoying now that I'd like to put an AlexPosn in my state!

I do feel like you're right about the bad interaction with whatever TemplateHaskell is producing for the lenses. I will try and investigate a little, though I am not very familiar with TemplateHaskell myself.

[remexre]

Did anyone figure anything out here? I'm in the same situation (with a virtually identical UserState, too)

[Ptival]

@remexre I personally ended up writing the lenses manually:

https://github.com/Ptival/language-ocaml/blob/f7865ae8881cc4934705935e402433ec951fdb49/lib/Language/OCaml/Parser/Generator/Lexer.x#L585-L595

[poscat0x04]

This also affects optics-th.

[andreasabel]

Could such a problem maybe solved by a small Alex.Types library that exports the types Alex uses/defines? Then you could define your AlexUserState and its lenses outside of the .x file.

[andreasabel]

Could such a problem maybe solved by a small Alex.Types library that exports the types Alex uses/defines?

Atm, alex does not come with a library component, so it could be placed there. Of course, this would be a major change to the usage of alex since packages would to have to depend on the alex library also, not just on the build-tool.

An advantage is that if you have several Alex-generated lexers in a project, they would share some code rather than duplicating it for each of these lexers.

[andreasabel]

@Ptival wrote:

But, the moment I add the line:

makeLenses ''AlexUserState

I get a bunch of errors:

Once you are using TemplateHaskell, the order of declarations in file matters, because the TH expressions are evaluated top-to-bottom.
(Note that there are also tricks like inserting return [] in the file before invoking TH code, to force the scope checker to finish the part above the return [] before running TH stuff.)

Are you putting your definitions in the prologue or epilogue? Intuitively, the epilogue should work better. Since this issue report lacks a MWE, I cannot say much more at this point.

CC @remexre @poscat0x04

[Ptival]

Supposedly (I posted this 5 years ago), I was putting it at the exact same location that I put the hand-written alternative here

[andreasabel]

Ok, the problem is that the generated .hs file looks like this:

1. user prologue
2. generated code: Alex wrappers
3. user epilogue
4. generated code: alex_actions, main lexing routine. This contains data types like AlexError

Maybe the problem can be fixed by swapping 3. and 4, so that the epilogue code comes last in the generated file. I do not know the reasons for the current ordering, @simonmar or @Ericson2314 , do you know?

[andreasabel]

Maybe the problem can be fixed by swapping 3. and 4, so that the epilogue code comes last in the generated file.

This seems to do the job, see:

• #225

[andreasabel]

@Ptival : Could you test PR #225 whether it solves your problem?

[Ptival]

Tried today but I can neither build nor install alex from source.

• cabal build fails with:

Resolving dependencies...
Build profile: -w ghc-9.2.7 -O1
In order, the following will be built (use -v for more details):
 - alex-3.2.7.3 (exe:alex) (first run)
Preprocessing executable 'alex' for alex-3.2.7.3..
Error: cabal-3.10.1.0: The program 'happy' is required but it could not be
found

Error: cabal: Failed to build exe:alex from alex-3.2.7.3.

Even though I have alex and happy installed, in my $PATH.

• cabal install fails with:

Error: cabal: sdist of alex-3.2.7.3: filepath wildcard 'src/Parser.y.boot'
does not match any files.

• cabal install ./dist-newstyle/sdist/alex*.tar.gz fails with:

Error: cabal: Unknown target './dist-newstyle/sdist/alex-3.2.7.3.tar.gz'.
The package alex has no file target 'dist-newstyle/sdist/alex-3.2.7.3.tar.gz'.

---

Tested with: GHC 9.2.7 obtained via GHCup cabal-install 3.10.1.0 alex 3.2.7.3 happy 1.20.1.1

[andreasabel]

@Ptival: Yes unfortunately, you need a special install.

make sdist

This produces a tarball, and then you install it with

cabal v1-install ./dist-newstyle/sdist/alex*.tar.gz 

(Note the use of the old command v1-install!).

[andreasabel]

I am going ahead with #225 now.

[andreasabel]

Fix released in 3.2.7.4.