How to know how each layer is used?

[yukiarimo]

Hello! If I send a raw text into the model like "Hi there", how to know which layer in percentage is used to determine the most responsible one for this prompt (would like a single script, NOT USING TRANSFORMERLENS)?! Thanks.

[Katehuuh]

Mergekit doesn't have a direct prompts per-layer analysis; But this might help:

• positive_prompts: from mergekit-moe is closest; it allows creating MoE models where different experts handle specific types of prompts:
  https://github.com/arcee-ai/mergekit/blob/93b7693a6940afa7f45ef9f676098747b5883fa4/docs/moe.md?plain=1#L19-L20

• As for dense... it doesn't directly use prompts but you can try mergekit-evolve of docs/evolve.md to test different merges and find the best parameters by evaluates:
  https://github.com/arcee-ai/mergekit/blob/93b7693a6940afa7f45ef9f676098747b5883fa4/docs/evolve.md?plain=1#L45-L46

---

@cg123 Can a method potentially use prompts to guide layer merging of dense models? For example, merge specialized math+coder models?

[yukiarimo]

Thanks! By the way, if I'll make a Gemma 3 12B to a MoE there will be no support of it (like in MLX/llama.cop), right? So, can't run :(

[yukiarimo]

Did fine-tuning (LoRA, no way for full, unfortunately).