gpt-neox icon indicating copy to clipboard operation
gpt-neox copied to clipboard

Add Support for Returning Logits for Generated Tokens

Open Kyle1668 opened this issue 2 years ago • 3 comments

Closes #588

Kyle1668 avatar Jul 01 '22 15:07 Kyle1668

The original issue mentioned potentially returning the log-likelihood of the tokens through some integration in the generate.py interface. I decided against implementing that support in this PR and instead returned the raw logits. My reasoning is that end users can then operate on these logits as needed.

Kyle1668 avatar Jul 03 '22 00:07 Kyle1668

Outputs from ./deepy.py generate.py ./configs/20B.yml ./configs/text_generation.yml on my branch

With return_logits: true

image

without

{
  "context": "",
  "text": "Entry DescriptionWild Agnes has released this extraordinary new",
  "length": 10,
  "finished": false,
  "message": null,
  "duration_seconds": 6.626634120941162
}

on main

{
  "context": "",
  "text": "Entry DescriptionWild Agnes has released this extraordinary new",
  "length": 10,
  "finished": false,
  "message": null,
  "duration_seconds": 6.523743629455566
}

Kyle1668 avatar Jul 03 '22 01:07 Kyle1668

I took a stab at writing some unit tests for this change but ran into an issue with mpu.get_model_parallel_src_rank() throwing an exception saying that I needed to configure init_process_group.

Kyle1668 avatar Jul 03 '22 01:07 Kyle1668