Training from scratch on a new dataset

[FSet89]

I have a dataset consisting on a single large point cloud. The point cloud has been split into a train file and validation file. Both are txt with X, Y, Z, R, G, B, L (label). I'm creating a custom class extending DefaultDataset and a custom configuration file. Is this correct? I see that other dataset classes have different methods. Which one should I take inspiration from to implement my own?

[Gofinge]

Hi, you can consider download one processed ScanNet dataset, and check the data structure.

[FSet89]

Thank you, I will. Do you think that I should split my training cloud into multiple file, or will this be managed by the loader? I'm not sure about the correct input dimension for the network.

[Gofinge]

Do you think that I should split my training cloud into multiple file

Typically no need for this.

will this be managed by the loader

Managed by dataset transforms.

the correct input dimension for the network.

Depend on your setting. (data.tranform.Collect.feat_key)

[FSet89]

Depend on your setting. (data.tranform.Collect.feat_key)

Can you explain this?

[Gofinge]

Here is an example: https://github.com/Pointcept/Pointcept/blob/main/configs/scannet/semseg-pt-v3m1-0-base.py#L134. Here, the feat_key is ["coord", "normal"], which means that the Collect function will create "feat" with "coord" and "normal" ([n, 6]), further resulting in an in-channel of 6.

[FSet89]

Here is an example: https://github.com/Pointcept/Pointcept/blob/main/configs/scannet/semseg-pt-v3m1-0-base.py#L134. Here, the feat_key is ["coord", "normal"], which means that the Collect function will create "feat" with "coord" and "normal" ([n, 6]), further resulting in an in-channel of 6.

Hi, thank you for the clarification. I understand that the second dimension depends on the features that you use. But what I'm not sure about is the first dimension: is there an optimal n? If so, I assume that it is better to split a large file into multiple ones.

[lihc-cz]

Here is an example: https://github.com/Pointcept/Pointcept/blob/main/configs/scannet/semseg-pt-v3m1-0-base.py#L134. Here, the feat_key is ["coord", "normal"], which means that the Collect function will create "feat" with "coord" and "normal" ([n, 6]), further resulting in an in-channel of 6.

Hi, thank you for the clarification. I understand that the second dimension depends on the features that you use. But what I'm not sure about is the first dimension: is there an optimal n? If so, I assume that it is better to split a large file into multiple ones.

Hello, I have the same question about the number of input point clouds. In the case of large number of point clouds, memory will be insufficient. There is a crop in the dataset transforms to reduce the number of point clouds of the data, but only part of the point clouds can be extracted. thank you.

[Gofinge]

Hi, as the crop is random, it can cover the whole point cloud with a larger epoch, and the randomness is good for the final result.

[lihc-cz]

Hi, as the crop is random, it can cover the whole point cloud with a larger epoch, and the randomness is good for the final result.

Thank you for replying to my query, I also want to know whether the setting of seed points will affect the random coverage of the whole point cloud. I am currently training on a single 4090 (24G), batch_size=1, and I see that seed = None in Pointceptmain\configs_base_\default_runtime.py, but set_seed(seed) in def default_setup(cfg): sets the seed randomly. For large-scale point clouds, setting random seeds, does the effect cover the whole point cloud in one epoch? Can the entire point cloud also be covered using random crop during training? like s3dis in pt_v3, the data in each val epoch is iterated 30 times. Does this cover the whole scene?

In other words, I found that after grid sampling and random crop, the indoor dataset basically covers the whole scene, but the outdoor dataset is too large, for example, 6 million points after 20 million + points using 0.04m sampling. Suppose that in random crop I set point_max=204800, epoch=30000, val_epoch=100, and 204800 points are extracted in each iteration. Then in a val_epoch, the same subcloud data list will be extracted 30 times, and in these 30 times, Are the randomly selected 204,800 points the same when given the random seed like above?

Looking forward to your reply!