[Feature] Add return hidden state in the native API

[zhaochenyang20]

Checklist

• [x] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• [x] 2. Please use English, otherwise it will be closed.

Motivation

JM is submitting a feature to get a hidden state. We can add examples at the beginning of the test file test/srt/test_hidden_states.py right now. Later rewrite this API and add it in the docs.

Try to add a native API instead of adding a parameter and relaunching the engine.

If anyone is interested in this, could reach out to me and try to get in touch.

Related resources

https://github.com/sgl-project/sglang/pull/3364