Matthew Waller comments

Results 37 comments of


                                            Matthew Waller

Begin simplifying CrossAttention so that it works better on the Apple Neural Engine

cc: @patrickvonplaten @pcuenca

Begin simplifying CrossAttention so that it works better on the Apple Neural Engine

Yeah, this is going to take more investigation. More experimenting has revealed that this may not be the exact pain point for ANE. I know that einsum can cause problems...

Converting a pipeline

For folks interested, I added a comment to this gist that has a bunch of the models converted. 3 out of 4! One bit I'm trying to convert is CLIP....

Converting a pipeline

Ha! The original author of the post was gracious enough to fix up that CLIPTextEncoder. Nice. One thing I'll need to tackle implementing to start experiment running on iPhone and...

Einsum unsupported equation format

I was just looking at that related to stable diffusions CrossAttention conversion. If support is added, if we be great to get a conversion without any transposes or reshapes. Those...

ValueError: Ops['identity_0', 'identity_0', 'identity_0', 'identity_0'] not found in block block0

Any guidance on what to do here? Running into the same problem

When to stop in the LLMEval?

@awni was working with phi34bit

When to stop in the LLMEval?

Alright, well unknownTokenId is 0 and eosTokenId is 32000, which I believe is correct, and it matches "eos_token": "", from HuggingFace. I can see in the debugger that the eosToken...

When to stop in the LLMEval?

Oh dang, yeah, I see that now, I pass in "\nWrite 2 words\n\n" after preparePrompt, and that should be 9 tokens or so. But it's encoded as 24 tokens!

When to stop in the LLMEval?

I made a little project where I directly looked for that token (32001) and returned .stop if I found it, in the LLMEvaluator. Once I did that, and got the...