tiktoken error on AWS lambda

[vicky141998]

I am using tiktoken in my project but when I deployed my project into AWS lambda I got the below error. Unknown encoding cl100k_base. Plugins found: ['tiktoken_ext.pycache', 'tiktoken_ext.openai_public']. how to solve this

[dtschleckser]

How is your Lambda layer set up? This looks like it could be an issue with your layer packaging & the way tiktoken scans for encodings. Can you try checking tiktoken_ext/openai_public to make sure ENCODING_CONSTRUCTORS is present and defines cl100k_base as you expect?

[vicky141998]

Hello Daniel Thanks for replying, I checked that file. cl100k_base is already present.

On Tue, Sep 26, 2023 at 9:52 AM Daniel Schleckser @.***> wrote:

How is your Lambda layer set up? This looks like it could be an issue with your layer packaging & the way tiktoken scans for encodings. Can you try checking tiktoken_ext/openai_public to make sure ENCODING_CONSTRUCTORS is present and defines cl100k_base as you expect?

— Reply to this email directly, view it on GitHub https://github.com/openai/tiktoken/issues/198#issuecomment-1734801270, or unsubscribe https://github.com/notifications/unsubscribe-auth/A52RQRUEDGJM3T2GUVX25PLX4JJ6VANCNFSM6AAAAAA5BB7QII . You are receiving this because you authored the thread.Message ID: @.***>

-- Regards,

Salesforce Developer Dreamwares IT Solution

Vicky Mahale +91 9028881205 [image: @.*** @.***> [image: phone-icon]https://dreamwares.com/ [image: phone-icon]Nashik, MH. (India)

[azmathr]

Did you fix it? Same issue

[vicky141998]

Not Solved

On Wed, Feb 14, 2024 at 11:51 AM azmathr @.***> wrote:

Did you fix it? Same issue

— Reply to this email directly, view it on GitHub https://github.com/openai/tiktoken/issues/198#issuecomment-1943151363, or unsubscribe https://github.com/notifications/unsubscribe-auth/A52RQRQCYE4DYFGTO2JFWYTYTRJWVAVCNFSM6AAAAAA5BB7QIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGE2TCMZWGM . You are receiving this because you authored the thread.Message ID: @.***>

-- Regards,

Salesforce Developer Dreamwares IT Solution

Vicky Mahale +91 9028881205 [image: @.*** @.***> [image: phone-icon]https://dreamwares.com/ [image: phone-icon]Nashik, MH. (India)

[azmathr]

@dtschleckser I get error in lambda while deploying through zappa

File "/var/task/summaryserver.py", line 13, in <module>
   encoding = tiktoken.get_encoding("cl100k_base")
 File "/var/task/tiktoken/registry.py", line 64, in get_encoding
   _find_constructors()
 File "/var/task/tiktoken/registry.py", line 44, in _find_constructors