Transform Messages Capability

[WaelKarkoub]

Why are these changes needed?

When I think of transforming chat messages (truncating, compressing, summarizing, etc.), it sounded to me like it's a transformation matrix problem in robotics. For example:

$\textbf{P}_2 = \textbf{R}_x \textbf{R}_y \textbf{R}_z \textbf{P}_1$

Where $\textbf{R}$ is the rotation transformation matrix relative to an axis, and $\textbf{P}$ is the position vector.

So I thought, why not approach transforming chat messages the same way?

$\textbf{M}_2 = \textbf{R}_a \textbf{R}_b \textbf{R}_c \textbf{M}_1$

For example:

• $\textbf{R}_a$ can be max number of messages,
• $\textbf{R}_b$ can be token limit,
• $\textbf{R}_c$ LLM Lingua compression

So I refactored TransformChatHistory in context_handling.py to get the same behavior described above, cc @gagb. So the ability is now called TransformMessages, and it accepts a list of the transformations you would like to apply to the messages. MessageTransform defines the interface of how the transform, $\textbf{R}$, should look like. I also implemented a few transforms as early examples (MaxMessagesTransform, TruncateMessageTransform)

Why is this useful?

There are some text-based use cases like I mentioned above that can be very helpful in managing the number of tokens used, but they can also be useful for other modalities. Imagine a certain LLM can only accept a certain image size, but the chat history has all kinds of image sizes and resolutions. All you have to do is create a transform (e.g. ResizeImageTransfrom) and pass it to the ability.

NOTE: Need to add tests

Related issue number

Checks

• [x] I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• [x] I've added tests (if relevant) corresponding to the changes introduced in this PR.
• [x] I've made sure all auto checks have passed.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 68.42105% with 36 lines in your changes are missing coverage. Please review.

Project coverage is 48.97%. Comparing base (78aa0eb) to head (8b0e77a).

Files	Patch %	Lines
...togen/agentchat/contrib/capabilities/transforms.py	76.82%	11 Missing and 8 partials :warning:
...entchat/contrib/capabilities/transform_messages.py	37.03%	17 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1923       +/-   ##
===========================================
+ Coverage   37.36%   48.97%   +11.61%     
===========================================
  Files          74       76        +2     
  Lines        7481     7591      +110     
  Branches     1617     1774      +157     
===========================================
+ Hits         2795     3718      +923     
+ Misses       4445     3559      -886     
- Partials      241      314       +73     

Flag	Coverage Δ
unittest	13.84% <64.03%> (?)
unittests	47.88% <4.38%> (+10.53%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[gagb]

This is nice! I'll check @dkirsche might also be interested

[gagb]

@WaelKarkoub I am running late on this PR-- I will have time after 14th to review this but I will do this. So sorry :)

[gagb]

@WaelKarkoub LFG!!! This align so much with what I am used to with other libs. Feedback:

1. Add NB demonstrating your code in use
2. Add test cases

Wdyt?

[WaelKarkoub]

@gagb notebook added :heavy_check_mark: tests added :heavy_check_mark:

[WaelKarkoub]

@gagb Thanks for the review! I updated the docstrs, let me know what you think of them

[gagb]

@gagb Thanks for the review! I updated the docstrs, let me know what you think of them

Will review today.

[BeibinLi]

This is a great PR. Thanks so much!

@sonichi @ekzhu @qingyun-wu @jackgerrits We had conversation about creating our own "message" format. Do you think we should proceed ASAP? This PR could be a good example converting TextMessage to ImageMessage or AudioMessage, etc.

For instance, we can have

class Message(Protocol):
     ...

class TextMessage:
    ...

class ImageMessage:
    ...

[WaelKarkoub]

@BeibinLi awesome idea! Yes, let us pause this PR till these new message formats go in, should make it easier to refactor for everyone. Let me know if I can help

[WaelKarkoub]

This is a great PR. Thanks so much!

@sonichi @ekzhu @qingyun-wu @jackgerrits We had conversation about creating our own "message" format. Do you think we should proceed ASAP? This PR could be a good example converting TextMessage to ImageMessage or AudioMessage, etc.

For instance, we can have

class Message(Protocol):
     ...

class TextMessage:
    ...

class ImageMessage:
    ...

@BeibinLi awesome idea! Yes, let us pause this PR till these new message formats go in, should make it easier to refactor for everyone. Let me know if I can help

@gagb @rickyloynd-microsoft I'll be pausing development till the proposed new message format goes in, should I send it back to draft?

[gagb]

This is a great PR. Thanks so much! @sonichi @ekzhu @qingyun-wu @jackgerrits We had conversation about creating our own "message" format. Do you think we should proceed ASAP? This PR could be a good example converting TextMessage to ImageMessage or AudioMessage, etc. For instance, we can have

class Message(Protocol):
     ...

class TextMessage:
    ...

class ImageMessage:
    ...

@BeibinLi awesome idea! Yes, let us pause this PR till these new message formats go in, should make it easier to refactor for everyone. Let me know if I can help

@gagb @rickyloynd-microsoft I'll be pausing development till the proposed new message format goes in, should I send it back to draft?

I need @sonichi help to make that decision. I am personally okay with it for following reasons:

• nobody is blocked
• if someone starts using your contribution and we introduced breaking changes, that'd create overhead to communicate On the other hand, if the message protocol blocks this PR for 2 months, that won't be ideal?

[rickyloynd-microsoft]

This is a great PR. Thanks so much! @sonichi @ekzhu @qingyun-wu @jackgerrits We had conversation about creating our own "message" format. Do you think we should proceed ASAP? This PR could be a good example converting TextMessage to ImageMessage or AudioMessage, etc. For instance, we can have

class Message(Protocol):
     ...

class TextMessage:
    ...

class ImageMessage:
    ...

@BeibinLi awesome idea! Yes, let us pause this PR till these new message formats go in, should make it easier to refactor for everyone. Let me know if I can help

@gagb @rickyloynd-microsoft I'll be pausing development till the proposed new message format goes in, should I send it back to draft?

I need @sonichi help to make that decision. I am personally okay with it for following reasons:

• nobody is blocked
• if someone starts using your contribution and we introduced breaking changes, that'd create overhead to communicate On the other hand, if the message protocol blocks this PR for 2 months, that won't be ideal?

I agree with @gagb. A change in the message protocol touches a great many things, and it's unclear when it will be coming. So unless there's some clear direction from @sonichi that all work involving the message protocol should stop and wait, I think this PR should go ahead.

[ekzhu]

I feel the same as @rickyloynd-microsoft and @gagb. A message protocol update can be a big update. It won't be ideal for this PR to be blocked by that.

[gagb]

@WaelKarkoub then it seems like other maintainers agree with proceeding with this PR. wdyt?

[WaelKarkoub]

@gagb sounds good!

@BeibinLi I just have a couple of unresolved comments and should be good to merge

[jackgerrits]

I think we should proceed with this PR (without addressing message types yet). I am working on at least solidifying the existing assumptions we have in place around message formats in a PR I hope to open soon. Once assumptions are solidified it will be clearer what the extensions we have in place are and how to type them

[sonichi]

@gagb could you confirm that the old way of handling context will be deprecated? Shall we remove the test and eventually remove the code and examples too? @jackgerrits there is one comment that needs your input.

[gagb]

@gagb could you confirm that the old way of handling context will be deprecated? Shall we remove the test and eventually remove the code and examples too? @jackgerrits there is one comment that needs your input.

Yes old way should be deprecated. The whole point of this solution was to generalize.

[WaelKarkoub]

@sonichi @jackgerrits updated the PR, notebook should be renderable and formatted to follow the guidelines