[core] Reduce unnecessary manifest reading during orphan clean

[bknbkn]

Purpose

Read manifest to find used files will cost time, if the candidates is empty, we don't need to do that

Tests

API and Format

Documentation