[GenAI] Introduce CausalLMPipelineChatClient for MEAI.IChatClient

[LittleLittleCloud]

We are excited to review your PR.

So we can do the best job, please check:

• [x] There's a descriptive title that will make sense to other developers some time from now.
• [x] There's associated issues. All PR's should have issue(s) associated - unless a trivial self-evident change such as fixing a typo. You can use the format Fixes #nnnn in your description to cause GitHub to automatically close the issue(s) when your PR is merged.
• [x] Your change description explains what the change does, why you chose your approach, and anything else that reviewers should know.
• [x] You have included any necessary tests in the same PR.

This PR implements CausalLMPipelineChatClient<TTokenizer, TCausalLMModel> class for MEAI.IChatClient's intergration.

This PR also refactors the AutoGen and semantic kernel intergretion for Phi3, which pulls out the prompt template parts into an individual Phi3ChatTemplateBuilder class.

This PR also update STJ to 8.0.5 to address a security warning for that package

The CausalLMPipelineChatClient is used as the abstract class for all IChatClient implementation in GenAI models. For now, the following implementations are available

• Phi3CausalLMChatClient
• Llama3CausalLMChatClient

The MistralCausalLMChatClient will come in the following PR with additional support for tool call.

Usage

Checkout the MEAI folder under Microsoft.ML.GenAI.Samples` for examples. The following examples are added

• Llama3_1
• Phi3

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 33.33333% with 152 lines in your changes missing coverage. Please review.

Project coverage is 68.84%. Comparing base (823fc17) to head (c5b9603). Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...rosoft.ML.GenAI.Core/CausalLMPipelineChatClient.cs	0.00%	48 Missing :warning:
...osoft.ML.GenAI.Phi/Phi3/Phi3ChatTemplateBuilder.cs	43.83%	35 Missing and 6 partials :warning:
...crosoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs	0.00%	24 Missing :warning:
...rosoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatClient.cs	0.00%	24 Missing :warning:
...soft.ML.GenAI.LLaMA/Llama3_1ChatTemplateBuilder.cs	69.23%	5 Missing and 3 partials :warning:
...c/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMAgent.cs	60.00%	6 Missing :warning:
...enAI.Phi/Phi3/Phi3CausalLMChatCompletionService.cs	83.33%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7270      +/-   ##
==========================================
+ Coverage   68.81%   68.84%   +0.03%     
==========================================
  Files        1461     1471      +10     
  Lines      272405   274124    +1719     
  Branches    28176    28411     +235     
==========================================
+ Hits       187442   188734    +1292     
- Misses      77727    78079     +352     
- Partials     7236     7311      +75     

Flag	Coverage Δ
Debug	68.84% <33.33%> (+0.03%)	:arrow_up:
production	63.30% <30.27%> (-0.01%)	:arrow_down:
test	89.18% <100.00%> (+0.10%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...enAI.Phi/Phi3/Phi3CausalLMTextGenerationService.cs	35.71% <100.00%> (+1.56%)	:arrow_up:
...st/Microsoft.ML.GenAI.LLaMA.Tests/LLaMA3_1Tests.cs	94.73% <100.00%> (+0.79%)	:arrow_up:
...enAI.Phi/Phi3/Phi3CausalLMChatCompletionService.cs	61.90% <83.33%> (-3.10%)	:arrow_down:
...c/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMAgent.cs	52.00% <60.00%> (-4.72%)	:arrow_down:
...soft.ML.GenAI.LLaMA/Llama3_1ChatTemplateBuilder.cs	68.83% <69.23%> (+0.20%)	:arrow_up:
...crosoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs	0.00% <0.00%> (ø)
...rosoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatClient.cs	0.00% <0.00%> (ø)
...osoft.ML.GenAI.Phi/Phi3/Phi3ChatTemplateBuilder.cs	43.83% <43.83%> (ø)
...rosoft.ML.GenAI.Core/CausalLMPipelineChatClient.cs	0.00% <0.00%> (ø)

... and 29 files with indirect coverage changes

[michaelgsharp]

/azp run

[azure-pipelines[bot]]

Azure Pipelines successfully started running 2 pipeline(s).

[michaelgsharp]

/backport to release/4.0

[github-actions[bot]]

Started backporting to release/4.0: https://github.com/dotnet/machinelearning/actions/runs/11749726947

[github-actions[bot]]

@michaelgsharp backporting to release/4.0 failed, the patch most likely resulted in conflicts:

$ git am --3way --empty=keep --ignore-whitespace --keep-non-patch changes.patch

Applying: leverage MEAI abstraction
Applying: Update src/Microsoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs
Applying: Update src/Microsoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs
Applying: Update src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatClient.cs
Applying: fix comments
Using index info to reconstruct a base tree...
M	src/Microsoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs
M	src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatClient.cs
Falling back to patching base and 3-way merge...
Auto-merging src/Microsoft.ML.GenAI.Phi/Phi3/Phi3CausalLMChatClient.cs
Auto-merging src/Microsoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs
CONFLICT (content): Merge conflict in src/Microsoft.ML.GenAI.LLaMA/Llama3CausalLMChatClient.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0005 fix comments
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

[github-actions[bot]]

@michaelgsharp an error occurred while backporting to release/4.0, please check the run log for details!

Error: git am failed, most likely due to a merge conflict.