Multiple predictions on one input

[JViggiani]

Hello. I'm exploring this locally in the demo jupyter.

I've modified this cell slightly here, adding some more information about Leonard Parker:

# Use Pretrained Model for Generation

from prediction import run_predict

texts = [
    "Leonard Parker studied physics and received his PhD from Harvard University in 1967. ",
    "Szewczyk played 37 times for Poland, scoring 3 goals .",
]
preds = run_predict(texts, path_checkpoint="cube_model")
preds.save("preds.json")
explore_data("preds.json")

However, for the first item in preds.json, I get:

{"tokens": ["Leonard", "Parker", "studied", "physics", "and", "received", "his", "PhD", "from", "Harvard", "University", "in", "@UNK@"], "entities": [{"span": [0, 2], "label": "Entity"}, {"span": [9, 11], "label": "Entity"}, {"span": [7, 8], "label": "Entity"}], "relations": [{"head": [0, 2], "tail": [9, 11], "label": "educated at", "qualifiers": [{"span": [7, 8], "label": "academic degree"}]}]}

Here some information was lost about what he studied (physics). To illustrate this more we can change the input:

Text: Freddy @UNK@ was a @UNK@ from the @UNK@ Leonard Parker studied physics and received his PhD from Harvard University in @UNK@ Tokens: ['Freddy', '@UNK@', 'was', 'a', '@UNK@', 'from', 'the', '@UNK@', 'Leonard', 'Parker', 'studied', 'physics', 'and', 'received', 'his', 'PhD', 'from', 'Harvard', 'University', 'in', '@UNK@'] Relation: head=(8, 10) tail=(17, 19) label='educated at' qualifiers=[Entity(span=(15, 16), label='academic degree')] Head: Leonard Parker, Relation: educated at, Tail: Harvard University Qualifier: academic degree, Value: PhD

It seems like the model is having issues with tokens it doesn't encounter during training. Has there been any progress in this area since?

Another thing I noticed was that it only ever produces a single output given an input. Is it possible to modify the model to produce some top-k outputs in any way?

Input (taken from hyperred training data):

"Napoleon Bonaparte was born in Ajaccio. Brazil is bordered by the countries of Argentina, Bolivia Colombia, and others."

Output:

Text: Napoleon Bonaparte was born in @UNK@ Brazil is bordered by the countries of @UNK@ Bolivia @UNK@ and @UNK@ Tokens: ['Napoleon', 'Bonaparte', 'was', 'born', 'in', '@UNK@', 'Brazil', 'is', 'bordered', 'by', 'the', 'countries', 'of', '@UNK@', 'Bolivia', '@UNK@', 'and', '@UNK@']

Text: @UNK@ played 37 times for @UNK@ scoring 3 goals . Tokens: ['@UNK@', 'played', '37', 'times', 'for', '@UNK@', 'scoring', '3', 'goals', '.'] Relation: head=(0, 1) tail=(5, 6) label='member of sports team' qualifiers=[Entity(span=(2, 3), label='number of matches played/races/starts'), Entity(span=(7, 8), label='number of points/goals/set scored')] Head: @UNK@, Relation: member of sports team, Tail: @UNK@ Qualifier: number of matches played/races/starts, Value: 37 Qualifier: number of points/goals/set scored, Value: 3

I'm surprised it produced no prediction and has UNKs given that the sentences were taken from the training data. Any idea why it might not make predictions here, and whether it might be possible to extend it to multiple predictions?