feat: Enhance research prompt to include documentation URLs for tasks and subtasks

[eyaltoledano]

"As a user, I want the research functionality to include direct URLs to the most current and relevant documentation, so that tools like Cursor can easily access up-to-date information during task implementation."

Achieve similar outcome to https://x.com/upstash/status/1910697488631181789

Motivation

Currently, the research feature leverages Perplexity AI to guide task generation effectively but doesn't explicitly provide direct access to current documentation. Including URLs to official and updated documentation will significantly improve the reliability and accuracy of implementation plans and enable seamless integration with tools such as Cursor that can automatically process and utilize these links when it reads the task or subtask details.

Proposed Solution

Adjust the Perplexity research prompt to explicitly request URLs of the latest, most relevant documentation. These URLs will be stored in a dedicated "Documentation URLs" section within each task or subtask, ensuring easy access for Cursor and other tools.

High-Level Workflow

1. Modify the Perplexity research prompt to explicitly ask for documentation URLs.
2. Capture and parse URLs provided by Perplexity's responses.
3. Automatically insert the URLs into a clearly labeled "Documentation URLs" section within tasks/subtasks.
4. Test integration to ensure Cursor and similar tools can automatically utilize these URLs.

Key Elements

• Research Prompt Enhancement:
  • Explicitly request documentation URLs from Perplexity
• Structured URL Storage:
  • Create and populate a new "Documentation URLs" section in tasks/subtasks
• Integration with Cursor:
  • Verify Cursor’s capability to automatically access and utilize provided URLs
  • Adjust system or user prompts or MCP tool descriptions as needed (or next_step objects)

Implementation Considerations

• Ensure reliability in URL extraction and validation => Pretty sure that Perplexity can return its research/reference URLs with its response
• Maintain clear, structured storage of documentation links within the task obejct
• Validate seamless integration and compatibility with Cursor and other IDEs
• Implement unit tests

Out of Scope (Future Considerations)

• Automatic content scraping or summarization from URLs
• Advanced validation of documentation accuracy beyond provided URLs

[ShreyPaharia]

@eyaltoledano I could pick this up. Seems like the output format for perplexity is:

{
  id: 'f78efa51-084d-48e8-a3a6-f71f5d1366e4',
  model: 'sonar-pro',
  created: 1745062360,
  usage: {
    prompt_tokens: 58,
    completion_tokens: 1075,
    total_tokens: 1133,
    search_context_size: 'low'
  },
  citations: [
    'https://www.anthropic.com/engineering/claude-code-best-practices',
    'https://support.anthropic.com/en/collections/9811458-api-usage-and-best-practices',
    'https://www.acorn.io/resources/learning-center/claude-api/',
    'https://www.anthropic.com/learn/build-with-claude',
    'https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview'
  ],
  object: 'chat.completion',
  choices: [
    {
      index: 0,
      finish_reason: 'stop',
      message: [Object],
      delta: [Object]
    }
  ]
}

So I could just pick up the citations and add it to the task documentsUrl directly. Could you confirm the approach ?

[ShreyPaharia]

@eyaltoledano can you confirm on the above. I think we could also update the prompt to specifically look at documentations. Can you assign this to me ?