HairNet AI

HairNet AI is a deep learning project with a goal of converting simple images into .obj 3D models. The 3D models are generated in Blender by creating central and particle mesh objects and generating a hair-type particle system. The neural network analyzes the image into parameters, from which a Blender Python (bpy) script generates the particle system and exports the object as a .obj file.

As it is difficult to determine the scale of an object from an image, HairNet aims to output an object that is approximately two meters across.

Input

The neural network takes in 200x200 pixel, grayscale png images, where the object is centered and occupies most of the frame.

Usage

To use the program, run the following command:

python hairnet.py input_image.png output_file_name.obj

This command uses the most recent iteration of the models. As of now, this command opens the generated object in Blender for debugging purposes. This command assumes you have Blender added to PATH. If you don't have Blender in the command line PATH, change the value of BLENDER_PATH in hairnet.py

Particle System Parameters

Each particle system has one central mesh and one particle mesh, where instances of the particle mesh are attached to the central mesh. The neural network outputs the following parameters:

• c_x: length of central object along x-axis
• c_y: length of central object along y-axis
• c_z: length of central object along z-axis
• c_offset: roundness of central object (higher means more round)
• p_l: length of particle object
• p_wh: width and height of particle object
• p_offset: roundness of particle object
• count: number of particle instances in the system
• scale_rand: randomness of particle size (on [0, 1]; higher means more random)

The generation script takes in these parameters and outputs an obj file.

Training Data

The training data for HairNet was generated by a script by randomly generating the parameter particle system parameters, central and particle materials (color and roughness), background color, lighting arrangement (by placing 3 lights on a sphere), and camera position (random position on a sphere around the particle system). The object was then rendered into an image using the EEVEE rendering engine and the parameters were saved to a csv file. In total, 6000 images were generated.

The p_wh value was generated using a normal/gaussian distribution in order to mimic more cactus and sea urchin shapes. If gaussian gaussian is switched to uniform, the generated images will look like timbits with giant sprinkles. While that shape still appears in the gaussian training data, it is more rare than the 'cactus' shape.

Current Approach

The current approach to designing the architecture of the artificial neural netowork (gen 3) is to create nine separate ANN's; one for each parameter. The previous approaches (gen 1 and gen 2) involved a single ANN that produced all nine parameters at once. However, the previous approaches failed because during the loss calculations, differences between sets (output and ground truth) would cancel out. As a result, the ANN optimized to reduce loss, but not match ground-truth values.

The current approach is not without its failures. The accuracy of the prediction of some parameters (c_x, c_y, p_l, and p_wh) moderately improved with thraining, while the prediction of other parameters did not noticably improve with training. Two possible factors may be at fault: improper data processing and improper architecture.

Prerequisites

• Blender 2.90 (bpy changes with each version)
• Python 2.82 (newer versions may break bpy)
• PyTorch
• opencv-python (for reading images)

Built With

• PyTorch
• Blender 2.90 (Blender Python API)
• Pandas
• JupyterLab

Contributors

• Alex Kalinins
• Parssa Kyanzadeh