amplify-category-api

amplify-category-api copied to clipboard

Environment information

System:
  OS: macOS 15.2
  CPU: (8) arm64 Apple M1 Pro
  Memory: 147.38 MB / 16.00 GB
  Shell: /bin/zsh
Binaries:
  Node: 22.0.0 - ~/.nvm/versions/node/v22.0.0/bin/node
  Yarn: 1.22.22 - /opt/homebrew/bin/yarn
  npm: 10.5.1 - ~/.nvm/versions/node/v22.0.0/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/auth-construct: 1.5.1
  @aws-amplify/backend: 1.13.0
  @aws-amplify/backend-auth: 1.4.2
  @aws-amplify/backend-cli: 1.4.6
  @aws-amplify/backend-data: 1.4.0
  @aws-amplify/backend-deployer: 1.1.13
  @aws-amplify/backend-function: 1.12.0
  @aws-amplify/backend-output-schemas: 1.4.0
  @aws-amplify/backend-output-storage: 1.1.4
  @aws-amplify/backend-secret: 1.1.5
  @aws-amplify/backend-storage: 1.2.4
  @aws-amplify/cli-core: 1.2.1
  @aws-amplify/client-config: 1.5.5
  @aws-amplify/deployed-backend-client: 1.5.0
  @aws-amplify/form-generator: 1.0.3
  @aws-amplify/model-generator: 1.0.12
  @aws-amplify/platform-core: 1.5.1
  @aws-amplify/plugin-types: 1.7.0
  @aws-amplify/sandbox: 1.2.9
  @aws-amplify/schema-generator: 1.2.6
  aws-amplify: 6.12.0
  aws-cdk: 2.174.1
  aws-cdk-lib: 2.172.0
  typescript: 5.5.4
No AWS environment variables
No CDK environment variables

Describe the bug

I am trying to generate content that is quite long. I never get responses that are 1000 words long using the prompt from below. The longest response I got was around 500 words and the shortest was 42 words long. I notice that from the moment I hit enter, 3 seconds elapse and then I get the response from the LLM. Whatever is generated during those 3 seconds (presumably before the execution of a lambda function times out?) is what I get. This makes the parameter maxTokens in inferenceConfiguration a bit pointless IMHO, unless I am missing something.

Reproduction steps

Schema

Storyteller: a
      .generation({
        aiModel: a.ai.model('Claude 3 Haiku'),
        inferenceConfiguration: {
          maxTokens: 4000,
        },
        systemPrompt:
          'You are a master storyteller. You write stories that are exactly 1000 words long on the topic provided.'
      })
      .arguments({ topic: a.string() })
      .returns(
        a.customType({
          content: a.string(),
        })
      )
      .authorization((allow) => allow.authenticated()),