PF-AFN
PF-AFN copied to clipboard
geyuying
How to make train_densepose file ？
You can download them in https://drive.google.com/file/d/1Uc0DTTkSfCPXDhd4CMx2TQlzlC6bDolK/view for VITON dataset. If you want to extract densepose for your own data, please refer to https://github.com/facebookresearch/DensePose.
I can get a similar effect using densepose_rcnn_R_50_FPN_s1x. but I don’t know how to save npy files like train_densepose
Follow the installation instructions on densepose. Then, use the following command to generate densepose annotations,
python2 tools/infer_simple.py
--cfg configs/DensePose_ResNet101_FPN_s1x-e2e.yaml
--output-dir DensePoseData/infer_out/
--image-ext [jpg or png]
--wts https://dl.fbaipublicfiles.com/densepose/DensePose_ResNet101_FPN_s1x-e2e.pkl
[Input image]
Hi @geyuying
The densepose repository at https://github.com/facebookresearch/DensePose does not provide the output as .npy files. So can you please let me know how to get .npy files for new images. Basically which numpy array are you saving ?
@mmm2016, @cloudy4next, @Zibin-Z , have you got any solutions for this on how to obtain the .npy files?
@soumyajit123dev just convert segmented image to numpy array and save as npy
@cloudy4next thanks for the reply.
I still have few doubts.
For a single input image, there are 2 png images produced by densepose. One is demo_im_INDS.png which is a grayscale image and the other is demo_im_IUV.png which is a RGB image. When you say segmented image, are you talking about the INDS one or the IUV one?
FYI, I have tried what you said already. I took some VITON images and implemented densepose on them. Then I converted the output png images to numpy array. Then I tried to compare this numpy array with the numpy array generated by the authors on the same VITON images. Unfortunately the two numpy arrays were different.
Any idea on this?
def load_dataset(): for item in dirs: if os.path.isfile(path+item): im = Image.open(path+item) print(im) im = np.array(np.uint8(im)) f, e = os.path.splitext(path+item) np.save("{}.npy".format(f),im)
if name == "main": load_dataset()
@soumyajit123dev
@cloudy4next how you ran dense pose can you help me I tried but failed in detection using cuda error
@cloudy4next I want to run DensePose to train PF-AFN algorithm, but not able to run that. Secondly How you use OPEN POSE to train PF-AFN? Are you storing the 18 Points as npy file?
@cloudy4next Could please explain this, what this notebook represents, There is no openpose running, correct me if I am wrong?
@geyuying What densepose checkpoint do you use? (e.g. densepose_rcnn_R_50_FPN_s1x, R_101_FPN_DL_s1x, and so on)
Hi everyone. I maybe find the solution. I get the npy file of the densepose results when I used detectron2.
Firstly, use the DensePose of the detectron2.
In my case, I use the densepose_rcnn_R_101_FPN_DL_s1x checkpoint.
Please download the checkpoint from this URL and deploy the downloaded file to the DensePose directory, i.e. ${your_directory}/detectron2/tree/master/projects/DensePose.
python apply_net.py dump configs/densepose_rcnn_R_101_FPN_DL_s1x.yaml model_final_844d15.pkl ${your_directory}/PF-AFN/PF-AFN_train/dataset/VITON_traindata/016809_0.jpg --output dump.pkl -v
Then, we can get the dump.pkl.
Next, we transform a densepose label result to a npy file.
Let start python.
import torch
import pickle
import numpy as np
import torch.nn.functional as F
with open("./dump.pkl", "rb") as f:
data = pickle.load(f)
im = data[0]["pred_densepose"][0].labels.to("cpu").reshape(1, 1, im.shape[0], im.shape[1]).to(torch.float32) # this tensor size is (1, 1, 253, 150)
im = F.interpolate(im, (256, 192)).reshape(256, 192) # nearest neighbor interpolation
im = im.numpy().astype(np.uint8)
np.save("./dump.npy", im)
Then, we get the npy file of the densepose labels. Anyway, I confirm the generated npy file is correct. Let start python.
import numpy as np
from PIL import Image
from skimage.color import label2rgb # if you don't install skimage, please install skimage using pip.
org = np.load("./016809_0.npy")
pred = np.load("./dump.npy")
org = label2rgb(org, bg_label=0)
pred = label2rgb(pred, bg_label=0)
pil_org = Image.fromarray(org)
pil_pred = Image.fromarray(pred)
pil_org.save("./org.png")
pil_pred.save("./pred.png")
Then, we get the label rgb image!! The results are as follows,
The target person image
The original PF-AFN densepose result
The densepose_rcnn_R_101_FPN_DL_s1x result
Note that this image size is (253, 150)
If you want to process many images, please see the usage of densepose.
Finally, all we need is what checkpoint does the author use. Thanks.
You can download them in https://drive.google.com/file/d/1Uc0DTTkSfCPXDhd4CMx2TQlzlC6bDolK/view for VITON dataset. If you want to extract densepose for your own data, please refer to https://github.com/facebookresearch/DensePose.
Hey, I have run this exact repo. I get two images, one IUV and one INDS image as ouptut. I have converted both the images to .npy files and compared them with the .npy file in the VITON dataset but they still don't match. Can you guide which densepose checkpoint or procedure did you use to get the numpy files?
def load_dataset(): for item in dirs: if os.path.isfile(path+item): im = Image.open(path+item) print(im) im = np.array(np.uint8(im)) f, e = os.path.splitext(path+item) np.save("{}.npy".format(f),im)
if name == "main": load_dataset()
@soumyajit123dev
The results still don't match with the ones that are provided in the viton dataset train_densepose folder.
Hi @maguro27, thank you for the solution. I've tried your solution, but I found that the element of org and pred array are different.... did you use these pred array to retrain the model already? How's the result ?
thank you !
Hi @soumyajit123dev, did you get the numpy file successfully? which image did you use, grayscale or RGB image?
thank you !
@spencerxhani Hi! Thank you for your response. I describe the details of how to make densepose outputs as follows. Note that you have to rewrite several parts of the detectron2 codes in this procedure. Hereafter, I talk about detectron2.
- Rewrite the
execute_on_outputsfunction of L.163 indetectron2/projects/DensePose/apply_net.py.
def execute_on_outputs(
cls: type, context: Dict[str, Any], entry: Dict[str, Any], outputs: Instances
):
image_fpath = entry["file_name"]
logger.info(f"Processing {image_fpath}")
result = {"file_name": image_fpath}
im = Image.open(image_fpath) # new line
im_size = im.size # new line
if outputs.has("scores"):
result["scores"] = outputs.get("scores").cpu()
if outputs.has("pred_boxes"):
result["pred_boxes_XYXY"] = outputs.get("pred_boxes").tensor.cpu()
if outputs.has("pred_densepose"):
if isinstance(outputs.pred_densepose, DensePoseChartPredictorOutput):
extractor = DensePoseResultExtractor()
elif isinstance(
outputs.pred_densepose, DensePoseEmbeddingPredictorOutput
):
extractor = DensePoseOutputsExtractor()
result["pred_densepose"] = extractor(outputs, im_size)[0] # changed line
context["results"].append(result)
- Rewrite the
DensePoseResultExtractorclass of L.86 indetectron2/projects/DensePose/densepose/vis/extractor.py.
class DensePoseResultExtractor(object):
"""
Extracts DensePose chart result with confidences from instances
"""
def __call__(
self, instances: Instances, im_size: tuple, select=None
) -> Tuple[Optional[DensePoseChartResultsWithConfidences], Optional[torch.Tensor]]: # changed line
if instances.has("pred_densepose") and instances.has("pred_boxes"):
dpout = instances.pred_densepose
boxes_xyxy = instances.pred_boxes
boxes_xywh = extract_boxes_xywh_from_instances(instances)
if select is not None:
dpout = dpout[select]
boxes_xyxy = boxes_xyxy[select]
converter = ToChartResultConverterWithConfidences()
results = [
converter.convert(dpout[i], boxes_xyxy[[i]], im_size) # changed line
for i in range(len(dpout))
]
return results, boxes_xywh
else:
return None, None
- Rewrite the
densepose_chart_predictor_output_to_result_with_confidencesfunction of L.159 indetectron2/projects/DensePose/densepose/converters/chart_output_to_chart_result.py.
def densepose_chart_predictor_output_to_result_with_confidences(
predictor_output: DensePoseChartPredictorOutput, boxes: Boxes, im_size: tuple
) -> DensePoseChartResultWithConfidences: # changed line
"""
Convert densepose chart predictor outputs to results
Args:
predictor_output (DensePoseChartPredictorOutput): DensePose predictor
output with confidences to be converted to results, must contain only 1 output
boxes (Boxes): bounding box that corresponds to the predictor output,
must contain only 1 bounding box
Return:
DensePose chart-based result with confidences (DensePoseChartResultWithConfidences)
"""
assert len(predictor_output) == 1 and len(boxes) == 1, (
f"Predictor output to result conversion can operate only single outputs"
f", got {len(predictor_output)} predictor outputs and {len(boxes)} boxes"
)
boxes_xyxy_abs = boxes.tensor.clone()
boxes_xywh_abs = BoxMode.convert(boxes_xyxy_abs, BoxMode.XYXY_ABS, BoxMode.XYWH_ABS)
box_xywh = make_int_box(boxes_xywh_abs[0])
labels = resample_fine_and_coarse_segm_to_bbox(
predictor_output, box_xywh, im_size
).squeeze(0) #changed line
uv = resample_uv_to_bbox(predictor_output, labels, box_xywh, im_size) # changed line
confidences = resample_confidences_to_bbox(predictor_output, labels, box_xywh)
- Rewrite the
resample_fine_and_coarse_segm_tensors_to_bboxfunction of L.32 indetectron2/projects/DensePose/densepose/converters/segm_to_mask.py.
def resample_fine_and_coarse_segm_tensors_to_bbox(
fine_segm: torch.Tensor,
coarse_segm: torch.Tensor,
box_xywh_abs: IntTupleBox,
im_size: tuple,
): # changed line
"""
Resample fine and coarse segmentation tensors to the given
bounding box and derive labels for each pixel of the bounding box
Args:
fine_segm: float tensor of shape [1, C, Hout, Wout]
coarse_segm: float tensor of shape [1, K, Hout, Wout]
box_xywh_abs (tuple of 4 int): bounding box given by its upper-left
corner coordinates, width (W) and height (H)
Return:
Labels for each pixel of the bounding box, a long tensor of size [1, H, W]
"""
x, y, w, h = box_xywh_abs
w = max(int(w), 1)
h = max(int(h), 1)
# coarse segmentation
coarse_segm_bbox = F.interpolate(
coarse_segm, (h, w), mode="bilinear", align_corners=False
).argmax(dim=1)
# combined coarse and fine segmentation
labels = (
F.interpolate(fine_segm, (h, w), mode="bilinear", align_corners=False).argmax(
dim=1
)
* (coarse_segm_bbox > 0).long()
)
canvas = torch.zeros(1, im_size[1], im_size[0]) # new line
canvas[0, y : y + h, x : x + w] = labels # new line
return canvas # changed line
- Rewrite the
resample_uv_tensors_to_bboxfunction of L.18 indetectron2/projects/DensePose/densepose/converters/chart_output_to_chart_result.py.
def resample_uv_tensors_to_bbox(
u: torch.Tensor,
v: torch.Tensor,
labels: torch.Tensor,
box_xywh_abs: IntTupleBox,
im_size: tuple,
) -> torch.Tensor: # changed line
"""
Resamples U and V coordinate estimates for the given bounding box
Args:
u (tensor [1, C, H, W] of float): U coordinates
v (tensor [1, C, H, W] of float): V coordinates
labels (tensor [H, W] of long): labels obtained by resampling segmentation
outputs for the given bounding box
box_xywh_abs (tuple of 4 int): bounding box that corresponds to predictor outputs
Return:
Resampled U and V coordinates - a tensor [2, H, W] of float
"""
x, y, w, h = box_xywh_abs
w = max(int(w), 1)
h = max(int(h), 1)
u_bbox = F.interpolate(u, (h, w), mode="bilinear", align_corners=False)
v_bbox = F.interpolate(v, (h, w), mode="bilinear", align_corners=False)
canvas_u = torch.zeros(1, 25, im_size[1], im_size[0], device=u.device) # new line
canvas_v = torch.zeros(1, 25, im_size[1], im_size[0], device=v.device) # new line
canvas_u[:, :, y : y + h, x : x + w] = u_bbox # new line
canvas_v[:, :, y : y + h, x : x + w] = v_bbox # new line
uv = torch.zeros([2, im_size[1], im_size[0]], dtype=torch.float32, device=u.device) # changed line
for part_id in range(1, canvas_u.size(1)): # changed line
uv[0][labels == part_id] = canvas_u[0, part_id][labels == part_id] # changed line
uv[1][labels == part_id] = canvas_v[0, part_id][labels == part_id] # changed line
return uv
- Generate densepose outputs.
python apply_net.py dump \
configs/densepose_rcnn_R_101_FPN_DL_s1x.yaml \
model_final_844d15.pkl \
${dataset_directory_which_includes_images} \
--output dump.pkl \
-v
- Convert pickle file to numpy file. Note that If you want to convert other outputs (e.g. uv map), please change the L.15 to L.18.
import os
import sys
import pickle
import argparse
import numpy as np
from tqdm import tqdm
def main(opt):
os.makedirs(opt.save_dir, exist_ok=True)
with open(opt.pkl_file, "rb") as f:
datum = pickle.load(f)
for data in tqdm(datum):
im_name = os.path.splitext(os.path.basename(data["file_name"]))[0]
try:
im = data["pred_densepose"][0].labels
im = im.numpy().astype(np.uint8)
np.save(os.path.join(opt.save_dir, im_name + ".npy"), im)
except Exception as e:
print(e)
print(data)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--pkl_file", "-p", help="pickle file path")
parser.add_argument("--save_dir", "-s", help="save path")
opt = parser.parse_args()
main(opt)
- Then, you can get NumPy files of densepose outputs!
Thanks.
If I have .npy files already of images that are 512 by 384 in resolution, how can I get the .npy arrays for the images that are 256 by 192?
@mesllo Hi. One solution is downsampling .npy files using the nearest neighbor downsampling. However, this is a stopgap solution. Another solution is if you have the original image files, you should process them using detetron2.
@maguro27 Thank you for your help! now I can get densepose images sized original images not cropped images.
您可以下载它们https://drive.google.com/file/d/1Uc0DTTkSfCPXDhd4CMx2TQlzlC6bDolK/view用于 VITON 数据集。如果您想为自己的数据提取密集数据，请参阅https://github.com/facebookresearch/DensePose。
嘿，我已经运行了这个确切的存储库。我得到了两张图像，一张 IUV 和一张 INDS 图像作为 ouptut。我已经将两个图像都转换为 .npy 文件，并将它们与 VITON 数据集中的 .npy 文件进行了比较，但它们仍然不匹配。你能指导你使用哪个密集检查点或过程来获取 numpy 文件吗？
Hello, I also encountered a mismatch problem. Have you solved it？ @mesllo @mmm2016 @mrsahabu @geyuying @maguro27
@spencerxhani嗨！感谢您的回复。我描述如何进行密集输出的详细信息如下。请注意，在此过程中，您必须重写 detectron2 代码的几个部分。下面，我谈谈侦探2。
- 重写L.163的功能。
execute_on_outputs``detectron2/projects/DensePose/apply_net.pydef execute_on_outputs( cls: type, context: Dict[str, Any], entry: Dict[str, Any], outputs: Instances ): image_fpath = entry["file_name"] logger.info(f"Processing {image_fpath}") result = {"file_name": image_fpath} im = Image.open(image_fpath) # new line im_size = im.size # new line if outputs.has("scores"): result["scores"] = outputs.get("scores").cpu() if outputs.has("pred_boxes"): result["pred_boxes_XYXY"] = outputs.get("pred_boxes").tensor.cpu() if outputs.has("pred_densepose"): if isinstance(outputs.pred_densepose, DensePoseChartPredictorOutput): extractor = DensePoseResultExtractor() elif isinstance( outputs.pred_densepose, DensePoseEmbeddingPredictorOutput ): extractor = DensePoseOutputsExtractor() result["pred_densepose"] = extractor(outputs, im_size)[0] # changed line context["results"].append(result)
- 重写L.86的类。
DensePoseResultExtractor``detectron2/projects/DensePose/densepose/vis/extractor.pyclass DensePoseResultExtractor(object): """ Extracts DensePose chart result with confidences from instances """ def __call__( self, instances: Instances, im_size: tuple, select=None ) -> Tuple[Optional[DensePoseChartResultsWithConfidences], Optional[torch.Tensor]]: # changed line if instances.has("pred_densepose") and instances.has("pred_boxes"): dpout = instances.pred_densepose boxes_xyxy = instances.pred_boxes boxes_xywh = extract_boxes_xywh_from_instances(instances) if select is not None: dpout = dpout[select] boxes_xyxy = boxes_xyxy[select] converter = ToChartResultConverterWithConfidences() results = [ converter.convert(dpout[i], boxes_xyxy[[i]], im_size) # changed line for i in range(len(dpout)) ] return results, boxes_xywh else: return None, None
- 重写L.159的功能。
densepose_chart_predictor_output_to_result_with_confidences``detectron2/projects/DensePose/densepose/converters/chart_output_to_chart_result.pydef densepose_chart_predictor_output_to_result_with_confidences( predictor_output: DensePoseChartPredictorOutput, boxes: Boxes, im_size: tuple ) -> DensePoseChartResultWithConfidences: # changed line """ Convert densepose chart predictor outputs to results Args: predictor_output (DensePoseChartPredictorOutput): DensePose predictor output with confidences to be converted to results, must contain only 1 output boxes (Boxes): bounding box that corresponds to the predictor output, must contain only 1 bounding box Return: DensePose chart-based result with confidences (DensePoseChartResultWithConfidences) """ assert len(predictor_output) == 1 and len(boxes) == 1, ( f"Predictor output to result conversion can operate only single outputs" f", got {len(predictor_output)} predictor outputs and {len(boxes)} boxes" ) boxes_xyxy_abs = boxes.tensor.clone() boxes_xywh_abs = BoxMode.convert(boxes_xyxy_abs, BoxMode.XYXY_ABS, BoxMode.XYWH_ABS) box_xywh = make_int_box(boxes_xywh_abs[0]) labels = resample_fine_and_coarse_segm_to_bbox( predictor_output, box_xywh, im_size ).squeeze(0) #changed line uv = resample_uv_to_bbox(predictor_output, labels, box_xywh, im_size) # changed line confidences = resample_confidences_to_bbox(predictor_output, labels, box_xywh)
- 重写L.32的功能。
resample_fine_and_coarse_segm_tensors_to_bbox``detectron2/projects/DensePose/densepose/converters/segm_to_mask.pydef resample_fine_and_coarse_segm_tensors_to_bbox( fine_segm: torch.Tensor, coarse_segm: torch.Tensor, box_xywh_abs: IntTupleBox, im_size: tuple, ): # changed line """ Resample fine and coarse segmentation tensors to the given bounding box and derive labels for each pixel of the bounding box Args: fine_segm: float tensor of shape [1, C, Hout, Wout] coarse_segm: float tensor of shape [1, K, Hout, Wout] box_xywh_abs (tuple of 4 int): bounding box given by its upper-left corner coordinates, width (W) and height (H) Return: Labels for each pixel of the bounding box, a long tensor of size [1, H, W] """ x, y, w, h = box_xywh_abs w = max(int(w), 1) h = max(int(h), 1) # coarse segmentation coarse_segm_bbox = F.interpolate( coarse_segm, (h, w), mode="bilinear", align_corners=False ).argmax(dim=1) # combined coarse and fine segmentation labels = ( F.interpolate(fine_segm, (h, w), mode="bilinear", align_corners=False).argmax( dim=1 ) * (coarse_segm_bbox > 0).long() ) canvas = torch.zeros(1, im_size[1], im_size[0]) # new line canvas[0, y : y + h, x : x + w] = labels # new line return canvas # changed line
- 重写L.18的功能。
resample_uv_tensors_to_bbox``detectron2/projects/DensePose/densepose/converters/chart_output_to_chart_result.pydef resample_uv_tensors_to_bbox( u: torch.Tensor, v: torch.Tensor, labels: torch.Tensor, box_xywh_abs: IntTupleBox, im_size: tuple, ) -> torch.Tensor: # changed line """ Resamples U and V coordinate estimates for the given bounding box Args: u (tensor [1, C, H, W] of float): U coordinates v (tensor [1, C, H, W] of float): V coordinates labels (tensor [H, W] of long): labels obtained by resampling segmentation outputs for the given bounding box box_xywh_abs (tuple of 4 int): bounding box that corresponds to predictor outputs Return: Resampled U and V coordinates - a tensor [2, H, W] of float """ x, y, w, h = box_xywh_abs w = max(int(w), 1) h = max(int(h), 1) u_bbox = F.interpolate(u, (h, w), mode="bilinear", align_corners=False) v_bbox = F.interpolate(v, (h, w), mode="bilinear", align_corners=False) canvas_u = torch.zeros(1, 25, im_size[1], im_size[0], device=u.device) # new line canvas_v = torch.zeros(1, 25, im_size[1], im_size[0], device=v.device) # new line canvas_u[:, :, y : y + h, x : x + w] = u_bbox # new line canvas_v[:, :, y : y + h, x : x + w] = v_bbox # new line uv = torch.zeros([2, im_size[1], im_size[0]], dtype=torch.float32, device=u.device) # changed line for part_id in range(1, canvas_u.size(1)): # changed line uv[0][labels == part_id] = canvas_u[0, part_id][labels == part_id] # changed line uv[1][labels == part_id] = canvas_v[0, part_id][labels == part_id] # changed line return uv
- 生成密集输出。
python apply_net.py dump \ configs/densepose_rcnn_R_101_FPN_DL_s1x.yaml \ model_final_844d15.pkl \ ${dataset_directory_which_includes_images} \ --output dump.pkl \ -v
- 将泡菜文件转换为数字文件。请注意，如果要转换其他输出（例如uv贴图），请将L.15更改为L.18。
import os import sys import pickle import argparse import numpy as np from tqdm import tqdm def main(opt): os.makedirs(opt.save_dir, exist_ok=True) with open(opt.pkl_file, "rb") as f: datum = pickle.load(f) for data in tqdm(datum): im_name = os.path.splitext(os.path.basename(data["file_name"]))[0] try: im = data["pred_densepose"][0].labels im = im.numpy().astype(np.uint8) np.save(os.path.join(opt.save_dir, im_name + ".npy"), im) except Exception as e: print(e) print(data) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--pkl_file", "-p", help="pickle file path") parser.add_argument("--save_dir", "-s", help="save path") opt = parser.parse_args() main(opt)
- 然后，您可以获得密集输出的 NumPy 文件！
谢谢。
Hello, I converted them into. npy files and compared them with. npy files in the VITON dataset, but they still do not match. Have you solved the problem? What kind of densepose model does the author use？
@wang674 I cannot solve the mismatch problem. Hence, I used another checkpoint in my paper.
Could someone who has attempted the method proposed by @maguro27 please verify if the shape of data[0]["pred_densepose"][0].labels is indeed torch.Size([244, 82])? Alternatively, please let me know if I have overlooked something.
