boxo icon indicating copy to clipboard operation
boxo copied to clipboard

feat(example):add MFS-based UnixFS to CAR example

Open SAHU-01 opened this issue 1 month ago • 3 comments

Closes #663

Summary

Adds a example demonstrating how to create UnixFS DAGs and export them as CAR files using MFS (Mutable File System)

What's Included

✅ Proper Chunking (Not io.ReadAll)

  • Size-based: --chunker size-[bytes]
  • Rabin: --chunker rabin-[min]-[avg]-[max] (content-defined)
  • Buzhash: --chunker buzhash
  • Streams data, handles files of any size

✅ Directory Support

  • Recursive directory traversal
  • Preserves directory structure with subdirectories
  • Full filesystem tree import

✅ HAMT Sharding

  • Automatic HAMT for directories >174 files
  • Configurable threshold: --max-directory-links

✅ All Major ipfs add Flags

  • Chunking: --chunker, --raw-leaves
  • CID options: --cid-version, --hash, --inline, --inline-limit
  • Layout: --trickle (trickle DAG)
  • Metadata: --preserve-mode, --preserve-mtime

✅ MFS Integration

  • Uses mfs.NewRoot() to create filesystem
  • Uses mfs.PutNode() for adding nodes
  • Uses root.Flush() to persist changes

Example Usage

Basic File

cd examples/unixfs-to-car
go build

echo "Hello, IPFS!" > test.txt
./unixfs-to-car test.txt output.car

Output:

Root CID: bafybei...

Directory

mkdir mydir
echo "File 1" > mydir/file1.txt
echo "File 2" > mydir/file2.txt
./unixfs-to-car mydir output.car

With All Flags

./unixfs-to-car \
  --chunker size-1048576 \
  --raw-leaves \
  --hash blake2b-256 \
  --preserve-mode \
  --preserve-mtime \
  myfile.txt output.car

Testing

All 13 tests pass:

$ go test -v
=== RUN   TestBasicFileConversion
--- PASS: TestBasicFileConversion (0.00s)
=== RUN   TestLargeFileWithChunking
--- PASS: TestLargeFileWithChunking (0.01s)
=== RUN   TestDirectoryConversion
--- PASS: TestDirectoryConversion (0.00s)
=== RUN   TestRawLeaves
--- PASS: TestRawLeaves (0.00s)
=== RUN   TestTrickleDAG
--- PASS: TestTrickleDAG (0.00s)
=== RUN   TestMetadataPreservation
--- PASS: TestMetadataPreservation (0.00s)
=== RUN   TestDifferentHashFunctions
--- PASS: TestDifferentHashFunctions (0.00s)
=== RUN   TestDifferentChunkers
--- PASS: TestDifferentChunkers (0.01s)
=== RUN   TestInvalidHashFunction
--- PASS: TestInvalidHashFunction (0.00s)
=== RUN   TestInvalidChunker
--- PASS: TestInvalidChunker (0.00s)
=== RUN   TestNonExistentFile
--- PASS: TestNonExistentFile (0.00s)
=== RUN   TestLargeDirectoryWithHAMT
--- PASS: TestLargeDirectoryWithHAMT (0.03s)
PASS
ok      github.com/ipfs/boxo/examples/unixfs-to-car     0.287s

Tests cover:

  • ✅ Basic file conversion
  • ✅ Large file chunking (1MB+)
  • ✅ Directory conversion with subdirectories
  • ✅ Raw leaves
  • ✅ Trickle DAG layout
  • ✅ Metadata preservation
  • ✅ Multiple hash functions (SHA2-256, SHA2-512, Blake2b-256)
  • ✅ Multiple chunkers (size, rabin, buzhash)
  • ✅ HAMT sharding (200 files)
  • ✅ Error cases

Implementation Details

MFS-Based Workflow

// 1. Create MFS root
root, _ := mfs.NewRoot(ctx, dagService, emptyDir, nil, nil)

// 2. Import file/directory using UnixFS importer
node, _ := importFile(ctx, path, dagService, config, hashCode)

// 3. Add to MFS
mfs.PutNode(root, "/filename", node)
root.Flush()

// 4. Get CID and export to CAR
rootNode, _ := root.GetDirectory().GetNode()
exportToCAR(ctx, blockstore, rootNode.Cid(), outputPath)

This follows the same pattern as Kubo's ipfs add --to-files (see kubo/core/commands/add.go).

Screenrecording

image

https://github.com/user-attachments/assets/8bc86d41-52e7-4b31-adf3-aa890ad0875e

Documentation

README includes:

  • ✅ Usage examples for all features
  • ✅ All flags documented
  • ✅ Architecture explanation
  • ✅ Comparison to ipfs add
  • ✅ Performance characteristics
  • ✅ Code structure overview

@lidel this implementation satisfies all listed requirements and is ready for review!

SAHU-01 avatar Oct 20 '25 05:10 SAHU-01

Thank you for submitting this PR! A maintainer will be here shortly to review it. We are super grateful, but we are also overloaded! Help us by making sure that:

  • The context for this PR is clear, with relevant discussion, decisions and stakeholders linked/mentioned.

  • Your contribution itself is clear (code comments, self-review for the rest) and in its best form. Follow the code contribution guidelines if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment. Next steps:

  • A maintainer will triage and assign priority to this PR, commenting on any missing things and potentially assigning a reviewer for high priority items.

  • The PR gets reviews, discussed and approvals as needed.

  • The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution. We are very grateful for your contribution!

welcome[bot] avatar Oct 20 '25 05:10 welcome[bot]

It seems this issue might have been automatically generated. To help us address it effectively, please provide additional details.

We value the use of LLMs for code generation and welcome your contributions but please ensure your submission is of such quality that a maintainer will spend less time reviewing it than implementing it themselves. Verify the code functions correctly and meets our standards. If your change requires tests, kindly include them and ensure they pass.

If no further information is provided, the issue will be automatically closed in 7 days. Thank you for your understanding and for aiding us in maintaining quality contributions!

github-actions[bot] avatar Oct 21 '25 00:10 github-actions[bot]

Triage:

  • The example should be super concise and as short as possible. This is too verbose. We have helpers and wrappers that could have been used. Lots of flexibility used (cid versions etc. is out of the scope of an example).

hsanjuan avatar Oct 28 '25 15:10 hsanjuan