doctest fails to allocate memory

[ocharles]

I'm using cabal-doctest and the following whopper of a command line is built:

$ doctest -i -i/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.10.1/quoting-1.0.0/noopt/build/autogen -i/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.10.1/quoting-1.0.0/noopt/build -i/home/ollie/work/circuithub/quoting/lib -package-env=- -hide-all-packages -no-user-package-db -package-db=/home/ollie/.cabal/store/ghc-8.10.1/package.db -package-db=/home/ollie/work/circuithub/dist-newstyle/packagedb/ghc-8.10.1 -package-db=/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.10.1/quoting-1.0.0/noopt/package.conf.inplace -optP-include -optP/home/ollie/work/circuithub/dist-newstyle/build/x86_64-linux/ghc-8.10.1/quoting-1.0.0/noopt/build/autogen/cabal_macros.h -package-id=access-0.1.3-G4qeORg2IZ1GafQapKbA4B -package-id=adjunctions-4.4-3L855q73vEAQrHLCIrS0R -package-id=aeson-1.4.6.0-87MK9QIk6oMATdW4zwzTKX -package-id=base-4.14.0.0 -package-id=cassava-0.5.2.0-JcEwJBq6aYkrpZqgq3KPD -package-id=ch-model-0.1.0.0-inplace -package-id=ch-persistence-1.0-inplace -package-id=ch-servant-auth-1.0-inplace -package-id=circuithub-prelude-0.0.28-inplace -package-id=containers-0.6.2.1 -package-id=distributive-0.6.2-93n9twDTAloDDPaChLm5Hz -package-id=errors-2.3.0-9kOFLev5kiKLBSXK2tckls -package-id=generic-data-0.7.0.0-LEqvUGHM185DAaGF4RnsUI -package-id=logging-effect-1.3.10-8TOaXbUcSLeCxR67bNwCic -package-id=monoid-extras-0.5.1-LWyGpM1ZekmY3RmHcjPE1 -package-id=mtl-2.2.2 -package-id=opaleye-0.6.7004.2-3it6copuwOcL2cQHw7KtjU -package-id=postgresql-simple-0.6.2-BgvijHjeuFiC9yQsfWCDfX -package-id=prettyprinter-1.6.1-JYR0EmFtzfSIqvhE5QZWUw -package-id=rel8-0.1.0.0-8Hbiy9W133v7zpq9u12by7 -package-id=safe-decimal-0.2.0.0-xGHgc1q1sC8OixP6W4A52 -package-id=safe-money-0.9-IxlBNzRdZ728NvqpyR72zs -package-id=scientific-0.3.6.2-2JBA4boE6mq1raXLRr9Zpv -package-id=semialign-1.1-KwcTtoVfizZCsqIQASTcjy -package-id=semigroupoids-5.3.4-HlGah1tiXXN6YRVaskvnCK -package-id=servant-0.17-7nuWjWjgb4pBFYGPLrkjyx -package-id=servant-server-0.17-2TIRB2Fl1AN1AYAEOXqWQ1 -package-id=simple-effects-0.13.0.0-2gEBk4k6Nlt5D2X7ZxntcD -package-id=text-1.2.3.2 -package-id=these-1.1-AB0Dj8DOCcj6tnMeXDKj1G -package-id=time-1.9.3 -package-id=transformers-0.5.6.2 -package-id=units-2.4.1.2-HIwSZJjLV3a5YTFcDqfdo9 -package-id=units-defs-2.1.0.1-1SEMyvkhkcHJUCfGXLZVWc -package-id=unliftio-0.2.12-F6uxEi4gcaQFq5kk7ZQgGp -package-id=validation-1.1-DFjR9MdFrYL9xmDiFUozEi -package-id=witherable-0.3.5-IyPz4Gv0Jt2ibDuZg0u0F -package-id=base-compat-0.10.5-Gk0Pu7SQIuC58a2X5uclr -package-id=doctest-0.17-5Ex9GqTAwmrNlmqPGdCek -package=quoting-1.0.0 CircuitHub.Query.Quoting CircuitHub.Quoting CircuitHub.Quoting.AssemblyPricing.Composite CircuitHub.Quoting.AssemblyPricing.Composite.Features CircuitHub.Quoting.AssemblyPricing.Composite.Offer CircuitHub.Quoting.AssemblyPricing.Picofactory CircuitHub.Quoting.AssemblyPricing.Picofactory.Features CircuitHub.Quoting.AssemblyPricing.Picofactory2 CircuitHub.Quoting.AssemblyPricing.Picofactory2.Features CircuitHub.Quoting.AssemblyPricing.Regression CircuitHub.Quoting.AssemblyPricing.Regression.Features CircuitHub.Quoting.AssemblyPricing.Regression.Offer CircuitHub.Quoting.AssemblyPricing.Worthington CircuitHub.Quoting.Auth.Effect CircuitHub.Quoting.Auth.Effect.PostgreSQL CircuitHub.Quoting.Auth.Named CircuitHub.Quoting.Auth.Owns CircuitHub.Quoting.Auth.Proven CircuitHub.Quoting.Auth.RevisionReadAccess CircuitHub.Quoting.Auth.Some CircuitHub.Quoting.Auth.UserIsAdmin CircuitHub.Quoting.BoardPricing.Composite CircuitHub.Quoting.BoardPricing.Composite.Features CircuitHub.Quoting.BoardPricing.Composite.Offer CircuitHub.Quoting.BoardPricing.Manual CircuitHub.Quoting.BoardPricing.Manual.Features CircuitHub.Quoting.BoardPricing.Manual.Offer CircuitHub.Quoting.BoardPricing.OSHPark CircuitHub.Quoting.BoardPricing.OSHPark.Features CircuitHub.Quoting.BoardPricing.OSHPark.LayerCount CircuitHub.Quoting.BoardPricing.OSHPark.Offer CircuitHub.Quoting.BoardPricing.OSHPark.PricingOption CircuitHub.Quoting.BoardPricing.OSHPark.Shipping CircuitHub.Quoting.BoardPricing.RoyalCircuits CircuitHub.Quoting.BoardPricing.RoyalCircuits.Features CircuitHub.Quoting.BoardPricing.RoyalCircuits.LayerCount CircuitHub.Quoting.BoardPricing.RoyalCircuits.Offer CircuitHub.Quoting.BoardPricing.RoyalCircuits.SolderMaskColor CircuitHub.Quoting.BoardPricing.Storm CircuitHub.Quoting.BoardPricing.Storm.Config CircuitHub.Quoting.BoardPricing.Storm.Features CircuitHub.Quoting.BoardPricing.Storm.LayerCount CircuitHub.Quoting.BoardPricing.Storm.Offer CircuitHub.Quoting.CompositeQuoter CircuitHub.Quoting.CompositeQuoter.ExternalWork CircuitHub.Quoting.CompositeQuoter.Features CircuitHub.Quoting.LeadTime CircuitHub.Quoting.Markup CircuitHub.Quoting.PartPricing CircuitHub.Quoting.PartPricing.BomFeatures CircuitHub.Quoting.PartPricing.Features CircuitHub.Quoting.PartPricing.SourceOffer CircuitHub.Quoting.PartPricing.SourceOffer.Consigned CircuitHub.Quoting.PartPricing.SourceOffer.Purchase CircuitHub.Quoting.PartPricing.SourceSelection CircuitHub.Quoting.PartPricing.SubstituteFeatures CircuitHub.Quoting.Projects.Solution CircuitHub.Quoting.Projects.Solution.BOM CircuitHub.Quoting.Projects.Solution.BOM.Excess CircuitHub.Quoting.Projects.Solution.Flag CircuitHub.Quoting.Quantity.Area CircuitHub.Quoting.Quantity.Density CircuitHub.Quoting.Quantity.Dollar CircuitHub.Quoting.Quantity.Length CircuitHub.Quoting.Quantity.Mass CircuitHub.Quoting.Quantity.NumberOf CircuitHub.Quoting.Quantity.Time CircuitHub.Quoting.Quantity.Volume CircuitHub.Quoting.Queries CircuitHub.Quoting.Queries.Features CircuitHub.Quoting.Queries.Quote CircuitHub.Quoting.Queries.QuoteAlreadyExists CircuitHub.Quoting.Queries.WorthingtonCustomer CircuitHub.Quoting.Quote CircuitHub.Quoting.Quote.CompleteOffer CircuitHub.Quoting.Quote.CompleteOffer.BomItem CircuitHub.Quoting.Quote.CustomerParams CircuitHub.Quoting.Quote.Offer CircuitHub.Quoting.Quote.Offer.BomItem CircuitHub.Quoting.QuoteOwner CircuitHub.Quoting.Server.API CircuitHub.Quoting.Server.API.Response.Quote CircuitHub.Quoting.Server.Handlers CircuitHub.Quoting.Server.ServerState Control.Concurrent.Deduplicate Data.MonotoneMap Data.Ratio.Extra Numeric.Decimal.Extra
doctest: mmap 131072 bytes at (nil): Cannot allocate memory
doctest: Try specifying an address with +RTS -xm<addr> -RTS
doctest: internal error: m32_allocator_init: Failed to map
    (GHC version 8.10.1 for x86_64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

If I call ghci with these exact same arguments, it works - so I guess this is something specific to doctest.

[quasicomputational]

Is there a handy way of trying to reproduce this locally? e.g., I assume that it's not just a long command line that's the trigger, otherwise lens would probably be being bitten by that.

[ocharles]

I'm afraid I don't have a minimal repro. My guess is that it's more the number of packages being loaded, rather than the length of the command line. I can get doctest to run if I run it on a single module, but only for some modules - other single module invocations produce the same error. It was the end of the day when I reported this issue, but I'll dig into it in more detail now.

[ocharles]

Ok, I weirdly found the problem. Here's the diff that I need to stop doctest crashing...

@@ -57,12 +57,6 @@ import qualified Data.MonotoneMap as MonotoneMap
 import Data.Ratio.Extra
 
 
-{-# ann module "HLint: ignore Evaluate" #-}
-
-
-{-# ann module "HLint: ignore Use let" #-}
-
-
 -- | Burden rates per role in $/hour
 rates :: Durations Rational
 rates =
``

That's right, it seems that module annotations are the source of the problem!

[ocharles]

Ok, I might be slightly wrong there. The diff above is from a different branch I'm working on, but it doesn't fix master. The good news is I have a minimal repro on our master branch. The following single module is enough to crash doctest. You will need a dependency on units:

{-# language TemplateHaskell #-}
{-# language TypeFamilies #-}

module Data.Units.US.Money where

-- units
import Data.Metrology
import Data.Metrology.TH


declareDimension "Cost"


declareCanonicalUnit "Dollar" [t| Cost |] ( Just "USD" )


type instance DefaultUnitOfDim Cost =
  Dollar


declareDerivedUnit "Cent" [t| Dollar |] ( 1 / 100 ) ( Just "cent" )

[ocharles]

It seems to be related to actually starting the TH interpreter. If I compile this module with -ddump-splices and inline all TH, I get:

{-# language TypeFamilies #-}

module Data.Units.US.Money where

-- units
import Data.Metrology


data Cost = Cost
instance Dimension Cost


data Dollar = Dollar
instance Unit Dollar where
  type BaseUnit Dollar = Canonical
  type DimOfUnit Dollar = Cost
instance Show Dollar where
  show _ = "USD"


type instance DefaultUnitOfDim Cost =
  Dollar


data Cent = Cent
instance Unit Cent where
  type BaseUnit Cent = Dollar
  conversionRatio _ = 1.0e-2
instance Show Cent where
  show _ = "cent"

Which doctest can happily evaluate.

[ocharles]

Ok, final update - the ann module stuff also has to removed on the master branch. Perhaps these are two different problems that just manifest in the same failure. But TL;DR:

• Removing Template Haskell from Data.Units.US.Money and
• Removing ANN pragmas

Allows doctest to run on this package.

[quasicomputational]

I had a go at reproducing this in https://github.com/quasicomputational/doctest-271 but cabal v2-test works fine with GHC 8.10.2.

If you've still got the reproducer handy, can you see if haddock works on it? doctest and haddock are actually quite similar: if haddock works, then it's some kind of doctest-specific bug, but if haddock fails it may ultimately be a GHC issue.

[ocharles]

I unfortunately don't have a reproducible case around anymore. I'll close this for now. Hopefully the trail above may help anyone else who runs into this!

[qnikst]

Seems I have this issue on ghc-8.10.3 on Linux (works on macOS). Is there any fix I can try to recipe how to debug it?

It's a huge project with TH and seems that not all the packages with TH fails, so I'm not sure if I can extract the minimal case.

[quasicomputational]

Try haddocking the project as above, and also manually going in with GHCi and trying some of the doctest examples. If either of those break, it's fundamentally going to be a GHC issue. Otherwise, I'm not sure there's much for it except lopping off leaf modules or removing doctests and seeing if things still break.