Adjusting 'Confidence' when Predicting on New Data

[data-overload]

I'm using the model with the provided weights on my own images for prediction. Is there a way to make the model more picky when it selects pixels?

As you can see from my images, the model confuses road/grass for buildings, and I would rather it miss buildings than have false positives like this.

For example, with YOLO you can filter out bounding boxes with low confidence scores, so I'm wondering if there's a similar feature with this type of model.