feat(server): parse ydoc to markdown

[fengmk2]

close AI-190

PR Dependency Tree

• PR #12812 👈
  • PR #12811

This tree was auto-generated by Charcoal

[coderabbitai[bot]]

Walkthrough

A new feature was added to support retrieving a document's complete markdown content via a backend API. This includes a new endpoint, utility functions for parsing document snapshots to markdown, test cases (unit and end-to-end), and snapshot files to verify output correctness. No existing exported entities were changed.

Changes

File(s)	Change Summary
.../src/core/utils/blocksuite.ts	Added parseDocToMarkdownFromDocSnapshot function for parsing Yjs doc snapshots to markdown.
.../src/core/doc/reader.ts	Added DocMarkdown interface and getDocMarkdown method to DocReader and its subclasses.
.../src/core/doc-service/controller.ts	Added /workspaces/:workspaceId/docs/:docId/markdown endpoint to return markdown content.
.../src/core/utils/tests/blocksute.spec.ts .../blocksute.spec.ts.md	Added a test and snapshot for parsing doc snapshot to markdown.
.../src/core/doc/tests/reader-from-database.spec.ts .../reader-from-database.spec.ts.md	Added tests and snapshots for getDocMarkdown in the database doc reader.
.../src/core/doc/tests/reader-from-rpc.spec.ts .../reader-from-rpc.spec.ts.md	Added tests and snapshots for getDocMarkdown in the RPC doc reader.
.../src/core/doc-service/tests/controller.spec.ts	Added/updated tests for content type and markdown retrieval in doc service controller.
.../src/tests/e2e/doc-service/controller.spec.ts .../controller.spec.ts.md	Added end-to-end tests and snapshots for the markdown retrieval API endpoint.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Controller
    participant DocReader
    participant Database
    participant RPCService

    Client->>Controller: GET /workspaces/:wsId/docs/:docId/markdown
    Controller->>DocReader: getDocMarkdown(wsId, docId)
    alt RPC DocReader
        DocReader->>RPCService: fetch markdown (wsId, docId)
        alt Success
            RPCService-->>DocReader: {title, markdown}
            DocReader-->>Controller: {title, markdown}
        else Failure (network, etc.)
            DocReader->>Database: getDocMarkdown(wsId, docId)
            Database-->>DocReader: {title, markdown} or null
            DocReader-->>Controller: {title, markdown} or null
        end
    else Database DocReader
        DocReader->>Database: getDocMarkdown(wsId, docId)
        Database-->>DocReader: {title, markdown} or null
        DocReader-->>Controller: {title, markdown} or null
    end
    alt Found
        Controller-->>Client: 200 OK, JSON {title, markdown}
    else Not found
        Controller-->>Client: 404 Not Found, error JSON
    end

Assessment against linked issues

Objective	Addressed	Explanation
Implement doc reader to retrieve complete markdown (AI-190)	✅
Markdown output must conform to specified format/rules (AI-190)	✅

Assessment against linked issues: Out-of-scope changes

No out-of-scope changes found.

Possibly related PRs

• toeverything/AFFiNE#12840: Introduces or moves the markdown parsing logic (parsePageDoc) used in this PR's new utilities and endpoints.
• toeverything/AFFiNE#12625: Adjusts document snapshot reading utilities, which are also extended in this PR for markdown parsing.

Suggested reviewers

• forehalo

Poem

In the warren where code bunnies dwell,
Markdown magic, we now compel!
Snapshots abound, tests hop in line,
New endpoints serve content just fine.
With every doc, a story to share—
🐇 Happy coding, everywhere!

---

📜 Recent review details

Configuration used: CodeRabbit UI Review profile: CHILL Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e4d26ebf3a3c9fe46ca8785a623f85ad91a9999f and 541a2ccf07e0ea2286e6e8c8bb578c1f25fba543.

⛔ Files ignored due to path filters (4)

• packages/backend/server/src/__tests__/e2e/doc-service/__snapshots__/controller.spec.ts.snap is excluded by !**/*.snap
• packages/backend/server/src/core/doc/__tests__/__snapshots__/reader-from-database.spec.ts.snap is excluded by !**/*.snap
• packages/backend/server/src/core/doc/__tests__/__snapshots__/reader-from-rpc.spec.ts.snap is excluded by !**/*.snap
• packages/backend/server/src/core/utils/__tests__/__snapshots__/blocksute.spec.ts.snap is excluded by !**/*.snap

📒 Files selected for processing (12)

• packages/backend/server/src/__tests__/e2e/doc-service/__snapshots__/controller.spec.ts.md (1 hunks)
• packages/backend/server/src/__tests__/e2e/doc-service/controller.spec.ts (1 hunks)
• packages/backend/server/src/core/doc-service/__tests__/controller.spec.ts (4 hunks)
• packages/backend/server/src/core/doc-service/controller.ts (1 hunks)
• packages/backend/server/src/core/doc/__tests__/__snapshots__/reader-from-database.spec.ts.md (1 hunks)
• packages/backend/server/src/core/doc/__tests__/__snapshots__/reader-from-rpc.spec.ts.md (1 hunks)
• packages/backend/server/src/core/doc/__tests__/reader-from-database.spec.ts (1 hunks)
• packages/backend/server/src/core/doc/__tests__/reader-from-rpc.spec.ts (3 hunks)
• packages/backend/server/src/core/doc/reader.ts (5 hunks)
• packages/backend/server/src/core/utils/__tests__/__snapshots__/blocksute.spec.ts.md (1 hunks)
• packages/backend/server/src/core/utils/__tests__/blocksute.spec.ts (2 hunks)
• packages/backend/server/src/core/utils/blocksuite.ts (2 hunks)

✅ Files skipped from review due to trivial changes (3)

• packages/backend/server/src/tests/e2e/doc-service/snapshots/controller.spec.ts.md
• packages/backend/server/src/core/doc/tests/snapshots/reader-from-database.spec.ts.md
• packages/backend/server/src/tests/e2e/doc-service/controller.spec.ts

🚧 Files skipped from review as they are similar to previous changes (9)

• packages/backend/server/src/core/utils/blocksuite.ts
• packages/backend/server/src/core/utils/tests/blocksute.spec.ts
• packages/backend/server/src/core/doc/tests/snapshots/reader-from-rpc.spec.ts.md
• packages/backend/server/src/core/utils/tests/snapshots/blocksute.spec.ts.md
• packages/backend/server/src/core/doc/tests/reader-from-rpc.spec.ts
• packages/backend/server/src/core/doc-service/controller.ts
• packages/backend/server/src/core/doc/tests/reader-from-database.spec.ts
• packages/backend/server/src/core/doc-service/tests/controller.spec.ts
• packages/backend/server/src/core/doc/reader.ts

⏰ Context from checks skipped due to timeout of 90000ms (56)

• GitHub Check: Unit Test (5)
• GitHub Check: Unit Test (2)
• GitHub Check: Unit Test (4)
• GitHub Check: Unit Test (3)
• GitHub Check: Unit Test (1)
• GitHub Check: Native Unit Test
• GitHub Check: test-build-mobile-app / build-android-web
• GitHub Check: test-build-mobile-app / build-ios-web
• GitHub Check: loom thread test
• GitHub Check: y-octo binding test on aarch64-pc-windows-msvc
• GitHub Check: fuzzing
• GitHub Check: E2E BlockSuite Test (9)
• GitHub Check: y-octo binding test on x86_64-pc-windows-msvc
• GitHub Check: E2E Test (6)
• GitHub Check: y-octo binding test on aarch64-apple-darwin
• GitHub Check: y-octo binding test on x86_64-apple-darwin
• GitHub Check: E2E BlockSuite Test (7)
• GitHub Check: E2E BlockSuite Test (6)
• GitHub Check: Run native tests
• GitHub Check: E2E BlockSuite Test (2)
• GitHub Check: E2E BlockSuite Test (10)
• GitHub Check: E2E BlockSuite Test (8)
• GitHub Check: E2E BlockSuite Test (5)
• GitHub Check: E2E BlockSuite Test (3)
• GitHub Check: E2E Test (7)
• GitHub Check: E2E BlockSuite Test (4)
• GitHub Check: E2E BlockSuite Test (1)
• GitHub Check: E2E Test (10)
• GitHub Check: E2E Test (9)
• GitHub Check: E2E Test (4)
• GitHub Check: E2E Test (2)
• GitHub Check: E2E Test (8)
• GitHub Check: E2E Test (5)
• GitHub Check: E2E Test (3)
• GitHub Check: E2E Mobile Test (3)
• GitHub Check: E2E Mobile Test (2)
• GitHub Check: E2E Test (1)
• GitHub Check: E2E Mobile Test (4)
• GitHub Check: E2E Mobile Test (5)
• GitHub Check: E2E Mobile Test (1)
• GitHub Check: Typecheck
• GitHub Check: E2E BlockSuite Cross Browser Test (2, webkit)
• GitHub Check: Build @affine/electron renderer
• GitHub Check: E2E BlockSuite Cross Browser Test (2, firefox)
• GitHub Check: E2E BlockSuite Cross Browser Test (1, webkit)
• GitHub Check: E2E BlockSuite Cross Browser Test (2, chromium)
• GitHub Check: E2E BlockSuite Cross Browser Test (1, chromium)
• GitHub Check: Build AFFiNE native (aarch64-pc-windows-msvc)
• GitHub Check: E2E BlockSuite Cross Browser Test (1, firefox)
• GitHub Check: Build AFFiNE native (x86_64-pc-windows-msvc)
• GitHub Check: Analyze (typescript, blocksuite)
• GitHub Check: Analyze (typescript, affine)
• GitHub Check: Build Server native
• GitHub Check: Analyze (javascript, affine)
• GitHub Check: Analyze (javascript, blocksuite)
• GitHub Check: Lint

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 84.61538% with 14 lines in your changes missing coverage. Please review.

Project coverage is 55.62%. Comparing base (899ffd1) to head (541a2cc). Report is 5 commits behind head on canary.

Files with missing lines	Patch %	Lines
packages/backend/server/src/core/doc/reader.ts	73.46%	13 Missing :warning:
...ckages/backend/server/src/core/utils/blocksuite.ts	96.42%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           canary   #12812      +/-   ##
==========================================
- Coverage   55.93%   55.62%   -0.32%     
==========================================
  Files        2658     2658              
  Lines      125982   126073      +91     
  Branches    20031    19968      -63     
==========================================
- Hits        70474    70124     -350     
- Misses      53172    53610     +438     
- Partials     2336     2339       +3     

Flag	Coverage Δ
server-test	78.88% <84.61%> (-0.79%)	:arrow_down:
unittest	31.67% <ø> (+<0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.