File content not loaded when used in vars dict

[kkarski]

Describe the bug When vars contain a complex map of nested values, reading content of files with file:// prefix doesn't seem to work.

To Reproduce Here, the prompt template is injected with the file path string, not the file content

tests:
  - vars:
      reporting_period:
        current:
          period: '2023-12-31'
        previous:
          period: '2024-02-15'
          report: file://data/mixed_report_tables.html.txt

Here, the "file_content" variable WILL contain the content of the file but only "Object object" is inected into prompt template @ "report".

tests:
  - vars:
      file_content: file://data/mixed_report_tables.html.txt
      reporting_period:
        current:
          period: '2023-12-31'
        previous:
          period: '2024-02-15'
          report: {{ file_content }}

Expected behavior The content of the file should be read into the variable reporting_period.previous.report.

Screenshots If applicable, add screenshots to help explain your problem.

System information (please complete the following information):

• Promptfoo version: "promptfoo": "^0.84.1"

Additional context Add any other context about the problem here.

[sklein12]

Thanks, we took a look and see your two issues:

1. You'd like to be able to reference and load a file:// in a nested object
2. There's a bug where an object isn't serialized properly when referenced in a nested variable

As a workaround for now, you could write a custom function in your supported language of choice to enumerate your vars.

We'll prioritize these appropriately.

[adityabharadwaj198]

@sklein12 @mldangelo I was looking at tackling this issue, is this still relevant? If so do you have any suggestions / background for me on where I could start with this?

[mldangelo]

Thanks for volunteering to take this! We would love help on this issue. The files below may be helpful:

# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
prompts:
  - 'Analyze this report: {{ reporting_period.previous.report }}'
providers:
  - echo
tests:
  # Test Case 1: File path in a nested variable
  - description: "Test with file path in nested variable structure"
    vars:
      reporting_period:
        current:
          period: '2023-12-31'
        previous:
          period: '2022-12-31'
          report: file://data/mixed_report_tables.txt

  # Test Case 2: Loading file content into top-level variable and referencing it
  - description: "Test with file loaded into top-level var and referenced in nested var"
    vars:
      file_content: file://data/mixed_report_tables.txt
      reporting_period:
        current:
          period: '2023-12-31'
        previous:
          period: '2022-12-31'
          report: "{{ file_content }}" 

with file data/mixed_report_tables.txt as

<h1>Sample Report</h1><p>This is a test report with some data for the year 2023.</p><table><tr><th>Month</th><th>Revenue</th></tr><tr><td>January</td><td>,000</td></tr><tr><td>February</td><td>,000</td></tr></table>

let me know if you get stuck or need additional pointers!