`PlutusTx.loadFromFile` can't load validator from file

[KristianBalaj]

Summary

I have the following code:

import Plutus.Script.Utils.V2.Typed.Scripts.Validators (UntypedValidator)
import PlutusTx qualified

compiledMyScript :: PlutusTx.CompiledCode (MyScriptParams -> UntypedValidator)
compiledMyScript = $$(PlutusTx.loadFromFile "data/script.txt")

And this is the contents of the data/script.txt file (ie. the validator's CBOR encoded in hex):

4e4d01000033222220051200120011

When I try and use a validator loaded from file it fails with the following type of error:

Exception: Failed to deserialise our own program! This is a bug, please report it. Caused by: TooMuchSpace (0x000000010124969a,S {currPtr = 0x00000001012419d7, usedBits = 6})

I've also tried placing PlutusCore to the file:

(program
  2.0.0
  (lam i_0 [ [ (builtin addInteger) (con integer 16) ] i_0 ])
)

But it also failed with the same error.

Both of the files above were textual but I've also tried binary representation of the PlutusCore posted above, the following are the bits in textual form, but I've provided the binary data in a binary file:

00000010 00000000 00000000 00100011
00110111 00000000 10010000 00010000
00000000 00001001

It failed with a different error:

Exception: Failed to deserialise our own program! This is a bug, please report it. Caused by: BadEncoding (0x000000010b5aa2fa,S {currPtr = 0x000000010b5aa2f3, usedBits = 7}) "usedBits /= 0"

Steps to reproduce the behavior

Described above - trying to find the correct use for the function PlutusTx.loadFromFile. I haven't managed to find correct format that the function is expecting.

Actual Result

Results with a runtime error.

Expected Result

I'm expecting such compiled code would behave the same as I would load a classical Haskell written PlutusTx code.

I'm curious what format does it expect and in case it's something different how could I get that from the CBOR hex, or if there's other function that would create CompiledCode from CBOR hex.

Describe the approach you would take to fix this

No response

System info

Plutus revision: f003b096dbc5abca9dbb74fbff847914dd39fa6b

[effectfully]

OK, so this is what the docs say:

   {-# OPTIONS_GHC -fplugin-opt PlutusTx.Plugin:dump-uplc #-}

This will dump the output to a temporary file with a name based on the module name. The filename will be printed to the console when compiling the source file. You can then move it to a more permanent location.

It can be read in conveniently with loadFromFile as an alternative to compile.

But I thought I'd try generating the file directly using our executable. First I put

(program
  2.0.0
  (lam i_0 [ [ (builtin addInteger) (con integer 16) ] i_0 ])
)

into a file named addInteger.uplc. Then I converted the contents of the file to flat-namedDeBruijn and put the result into a file:

cabal run uplc convert -- -i addInteger16.uplc --if textual -o addInteger16.flat-namedDeBruijn --of flat-namedDeBruijn

Then I checked that everything's OK by evaluating the file:

cabal run uplc evaluate -- -i addInteger16.flat-namedDeBruijn --if flat-namedDeBruijn

which gave me

(lam i_0_0 [ [ (builtin addInteger) (con integer 16) ] i_0_0 ])

i.e. the expected program.

But when I tried to use loadFromFile on addInteger16.flat-namedDeBruijn:

compiledMyScript :: CompiledCode (Integer -> Integer)
compiledMyScript = $$(loadFromFile "<absolute_path>/addInteger16.flat-namedDeBruijn")

it gave me a decoding error:

>>> getPlc compiledMyScript
*** Exception: Failed to deserialise our own program! This is a bug, please report it. Caused by: BadEncoding (0x00007f84a40040e7,S {currPtr = 0x00007f84a40040d3, usedBits = 7}) "Failed to decode a universe"

(if I rerun the command, I get a different currPtr and what the hex that follows BadEncoding BTW, but the same failure overall)

So I looked into the source code of PlutusTx.Plugin:

    uplcP <- flip runReaderT plcOpts $ PLC.compileProgram plcP
    dbP <- liftExcept $ traverseOf UPLC.progTerm UPLC.deBruijnTerm uplcP
    when (opts ^. posDumpUPlc) . liftIO $
        dumpFlat
            (UPLC.UnrestrictedProgram $ void dbP)
            "untyped PLC program"
            (moduleName ++ ".uplc.flat")

The format is indeed flat-namedDeBruijn, however that sneaky void in there replaces all annotations with (), meaning we can no longer use getPlc, since the latter assumes that the program is annotated with SrcSpans.

So I performed deserialization in a bit more manual manner:

SerializedCode bs _ _ = compiledMyScript

deserialized :: UPLC.Program UPLC.NamedDeBruijn UPLC.DefaultUni UPLC.DefaultFun ()
Right (UnrestrictedProgram deserialized) = unflat (BSL.fromStrict bs)

-- >>> pretty deserialized
-- (program 2.0.0 (lam i_0 [ [ (builtin addInteger) (con integer 16) ] i_0 ]))

Hence the answer to your question:

I'm curious what format does it expect and in case it's something different how could I get that from the CBOR hex, or if there's other function that would create CompiledCode from CBOR hex.

is:

1. it's flat, not CBOR
2. it's a UPLC program
3. with names being NamedDeBruijn
4. and annotations being ()

BTW, I think there's some confusion with versions. V2 is a ledger version, not a Plutus version (the latest version of Plutus is 1.1.0). Unless I'm very wrong about that.

Hope this helps.

@michaelpj please review my comment for any misinformation.

[effectfully]

@michaelpj please review my comment for any misinformation.

@michaelpj a gentle nag.

Also, does dump-uplc have to erase annotations? What if someone wants to, say, dump the UPLC of a script, then load it later and use it in our debugger?

[KristianBalaj]

So I performed deserialization in a bit more manual manner:

@effectfully How did you get the compiledMyScript then since it was giving you error, too?

I still can't manage to import a validator into Plutus and get around the BadEncoding error. I was trying to use multiple command line tools to convert between multiple formats and get to PlutusTx.loadFromFile success, but also unsuccessfully.

I've also tried these 2 other approaches.

Approach 1

I've also tried using the textEnvelope, when I export my validator using writeFileTextEnvelope and then try to import it using the following code:

import Ledger qualified
import Prelude qualified as P
import Plutus.V2.Ledger.Api qualified as V2
import Data.ByteString.Lazy qualified as LBS
import Data.ByteString.Short qualified as SBS
import Cardano.Api (AsType (AsPlutusScript, AsPlutusScriptV2), readFileTextEnvelope)
import Cardano.Api.Shelley (PlutusScript (..))
import Codec.Serialise (deserialise)

readPlutusValidator :: P.FilePath -> P.IO Ledger.Validator
readPlutusValidator = P.fmap Ledger.Validator . readPlutusScript

readPlutusScript :: P.FilePath -> P.IO V2.Script
readPlutusScript = P.fmap deserialise . readPlutusScriptByteString

readPlutusScriptByteString :: P.FilePath -> P.IO LBS.ByteString
readPlutusScriptByteString path = do
  eith <- readFileTextEnvelope (AsPlutusScript AsPlutusScriptV2) path
  case eith of
    Left err -> fail $ P.show err
    Right (PlutusScriptSerialised scriptSBS) -> P.pure . LBS.fromStrict $ SBS.fromShort scriptSBS

When I run the readPlutusValidator from the snippet on a text envelope like this:

{
  "type": "PlutusScriptV2",
  "description": "",
  "cborHex": "4e4d01000033222220051200120011"
}

I get the BadEncoding error

Exception: Failed to deserialise our own program! This is a bug, please report it. Caused by: BadEncoding (0x00000001043a530f,S {currPtr = 0x00000001043a457c, usedBits = 3}) "usedBits /= 0"

Approach 2

Next, I've tried yet another approach:

import Data.ByteString.Lazy qualified as LBS
import Data.ByteString.Short qualified as SBS
import Cardano.Binary qualified as CBOR
import Ledger qualified
import Codec.Serialise (deserialise)
import Data.ByteString.Base16 qualified as Base16
import Data.Text.Encoding qualified as Text

cborToScript :: ByteString -> Either CBOR.DecoderError Ledger.Script
cborToScript x = deserialise . LBS.fromStrict . SBS.fromShort <$> CBOR.decodeFull' x

cborToValidator :: Text -> Either P.String Ledger.Validator
cborToValidator a =
  either Left Right (Base16.decode $ Text.encodeUtf8 a)
    >>= \b ->
      either (Left . P.show) Right (cborToScript b)
        >>= pure . Ledger.Validator

and then call the:

cborToValidator "4e4d01000033222220051200120011"

I once again get the following BadEncoding error:

Exception: Failed to deserialise our own program! This is a bug, please report it. Caused by: BadEncoding (0x00000001018c530f,S {currPtr = 0x00000001018c457c, usedBits = 3}) "usedBits /= 0"

Wrap-up

I don't currently have a workaround 🤔

[effectfully]

How did you get the compiledMyScript then since it was giving you error, too?

It was giving me an error when I attempted to deserialize it incorrectly. When I deserialized it in the right format, it worked out.

I've also tried these 2 other approaches.

Both use CBOR as far as I can tell and I wrote earlier that loadFromScript assumes a flat-based encoding, which is very much not CBOR.

I've created a PR with the logic that I described in my previous message: #5364. Please take a look.

[KristianBalaj]

Both use CBOR as far as I can tell

Yeah you're right, that's irrelevant to the issue title. I was just trying to load a validator other way.

I've created a PR with the logic that I described in my previous message: https://github.com/input-output-hk/plutus/pull/5364. Please take a look.

Thanks, will try it out 🙌

[effectfully]

Yeah you're right, that's irrelevant to the issue title. I was just trying to load a validator other way.

Ah, sorry, I misread. You should be able to use these two functions to serialize and deserialize a UPLC program (and for serializing a CompiledCore you can use the function right above those two).

[KristianBalaj]

@effectfully Thank you very much for your effort 👏 I've managed to make it work with your help.

[effectfully]

Great!

I'm keeping the issue open, because I believe the situation with annotations being required to be sometimes SrcSpans and sometimes () is confusing and we should look into it as I'm sure more people will bump into this issue if we don't mitigate it somehow.

@michaelpj please take a look at this thread once you're back from vacationing.

[effectfully]

@michaelpj now that you're back, could you please take a look at the discussion here? We've sorted out the specific issue, however I feel like our current alignment of loadFromFile is confusing and we're going to get more reports like this one if we don't improve it somehow.

[effectfully]

@michaelpj now that you're back, could you please take a look at the discussion here? We've sorted out the specific issue, however I feel like our current alignment of loadFromFile is confusing and we're going to get more reports like this one if we don't improve it somehow.

@michaelpj gentle nag.

[effectfully]

@michaelpj now that you're back, could you please take a look at the discussion here? We've sorted out the specific issue, however I feel like our current alignment of loadFromFile is confusing and we're going to get more reports like this one if we don't improve it somehow.

@michaelpj a mildly less gentle nag.

[sourabhxyz]

@KristianBalaj @effectfully Could we use deserialised as mentioned here to get a variable of type CompiledCode (Integer -> Integer) so that we can apply parameter to this script?

[sourabhxyz]

I tried applying my parameter to the obtained deserialised UPLC via finalUplc = fromJust $ UPLC.applyProgram deserialised (PlutusTx.liftProgram plcVersion100 myAssetClass) but when I serialise my finalUplc using serialiseUPLC $ toNameless finalUplc, I get different script hash...