Writing the Letter M with Houdini and PyTorch:
A First Step Toward Real-Time Deformers
I’ve been working in Houdini for more than eight years as an FX Artist, so when Entagma released a series focusing on applying machine learning to Houdini and VFX, I was immediately intrigued. Knowing the depth of their knowledge and expertise in the VFX field, I was eager to dive in.
It’s been a pleasure learning from Christopher Kopic through the Machine Learning 101 series and applying these insights to my own projects. This post is part of that learning journey, where I explore how machine learning can enhance Houdini workflows.
And yes—using Houdini and PyTorch to write the letter M may not seem groundbreaking. There are certainly easier ways to do it. However, this is just the first step toward something much more practical: creating real-time muscle and cloth deformers.
Why Real-Time Deformers?
Muscle and cloth simulations in Houdini are incredibly powerful tools for creating realistic deformations and motion. However, these simulations can be time-consuming—especially for characters used in multiple scenes or recurring projects like TV shows or commercials.
This is where machine learning can step in:
A trained neural network can perform the deformation almost instantaneously, providing a real-time preview for animators during the preliminary stages.
In some cases, the simulation can even be skipped entirely, relying on inference to deliver high-quality deformations.
Perfect for games: This solution is ideal for real-time applications like games, where time-consuming simulations aren’t feasible, but inference can provide very high fidelity and, most importantly, real-time results.
While this approach has its limitations—neural networks lack temporal awareness, meaning they don’t account for time-dependent factors like bounce and stretch —it provides much greater detail and accuracy than traditional weight painting. For more static tasks or scenarios where motion dynamics are less critical, the method offers a fast and effective alternative. So toned muscles and tight clothes are a perfect target to replace with inference, as their deformations are more predictable and less influenced by external forces like gravity or rapid motion.
Starting Small: Writing an M
To ensure the pipeline works as expected before diving into complex muscle deformation, I started with something simple: teaching a neural network to write the letter M. The goal here wasn’t to reinvent typography but to test and validate the entire workflow, from data creation in Houdini to training and inference in PyTorch.
The Workflow
1. Data Generation in Houdini
In Houdini, I scattered 50.000 points on a plane, each with:
Position data: The point’s coordinates in 3D space.
Color values: Assigned to form the shape of the letter M.
2. Exporting Data
This dataset was exported for use in PyTorch.
3. Training the Neural Network
Using PyTorch, I implemented a neural network to learn the regression task:
Predict the correct color value for each point based on its position.
Infer the “M” by marking points for coloring during inference.
The model took about 650 epochs to achieve high accuracy, with training visualized using Tensorboard.
4. Integration with Houdini via ONNX
Once trained, the model was exported using ONNX and re-integrated into Houdini. This ensures the network can be used directly in Houdini workflows for inference.
Using the trained model in Houdini
This short Houdini flipbook visualises the results of the inference, showcasing how the model progressively improves at writing the letter ‘M’ from epoch 50 to 650. The transformation highlights the learning process, with each iteration refining the accuracy of the output.
Technologies at Work
This project combines some of the most powerful tools and technologies in both the VFX and machine learning worlds:
• Houdini: As the backbone of this project, Houdini is used to generate training data. Its procedural nature and powerful simulation tools make it ideal for creating the complex datasets required for training machine learning models.
• PyTorch: The neural network was built and trained using PyTorch, one of the most popular frameworks for machine learning. Its flexibility and support for custom models made it the perfect choice for experimenting with regression tasks and preparing for more advanced deformations.
• ONNX (Open Neural Network Exchange): ONNX was used to export the trained neural network, ensuring seamless integration back into Houdini. This technology allows for smooth deployment of the model without compromising on performance or accuracy.
• Tensorboard: Tensorboard played a crucial role in monitoring the training process. It provided visualizations of the network’s learning curve, making it easy to track progress over the 650 epochs required to perfect the model.
This combination of tools ensures not only a robust and flexible workflow but also opens up new possibilities for applying machine learning directly within VFX pipelines.
The Road Ahead
While writing the letter M might not seem revolutionary, it represents a critical milestone:
The export/import pipeline is fully functional.
The neural network is successfully trained and integrated with Houdini.
The next step is to apply this workflow to muscle deformation. By leveraging machine learning, I aim to create a deformer that provides real-time deformations with comparable detail and accuracy to Houdini simulations, but at a fraction of the computational cost. This approach could greatly enhance efficiency for recurring projects and high-demand workflows.
Acknowledgements
A special thanks to Christopher Kopic and the team at Entagma for their invaluable “Machine Learning 101” series, which made this project possible.
Stay tuned for the next post, where I’ll dive into training the neural network to handle complex muscle deformation tasks.