Text-to-animation solution

T2M-GPT integration for human motion animation, an overview of a successful project.

Abto Software’s engineering team was contracted to design and deliver a unique text-to-animation solution. The objective was focused on empowering game developers to generate humanoid animations by providing detailed descriptions.

The integrated T2M-GPT model can process and convert textual instructions into in-depth motion sequences, thus allowing to create immersive and interactive animations.

Game development text-to-animation solution: Main goals

Our client is specializing in providing advanced solutions that change game development.

With modern-day large language model applications gaining momentum, we found the opportunity to utilize the algorithms to generate humanoid animations and replace motion capture to eliminate several bottlenecks. Having confidence in the promising potential and feasibility of leveraging large language models successfully, we suggested a minimum viable product to assess the concept’s maturity level.

Our team has covered the investigation of existing text-to-animation models and further T2M-GPT integration, as well as custom ML design and training to facilitate the outputs.

The project’s key objectives:

• Offer users the opportunity to harness large language model capabilities
• Successfully commercialize LLMs application to ensure market differentiation and competitiveness

Game development text-to-animation solution: Deep dive

With our text-to-animation solution, the users can provide detailed descriptions to generate their animations. No matter whether conversing, walking, running, fighting, shooting, swinging axes, the solution can reproduce any behavior with accuracy.

And this is how it works:

• First, the ML service is triggered to generate an animation by following user prompts
• Second, the T2M-GPT model is enabled to process the prompt and generate the animation in point cloud format
• The animation is converted into the BVH and FBX formats:
• The latter is rigged with the optimal rig for the associated mesh
• A custom ML model is applied to identify any errors and corruption
• Post-processing algorithms are applied to resolve found inconsistencies

The project in detail: Our contribution

Working on this project, we covered:

• Business analysis
• Software development
• ML training and integration
• 3D visualization environment development
• Thorough testing (manual testing, unit testing)
• Technical support and maintenance

Tech stack: React.js, Three.js, Python, JavaScript, and Django
ML stack: Python, PyTorch, NumPy, Matplotlib, Pandas, Scikit-learn

The project in detail: Key challenges

Working on this project, we faced several problems:

Output conversion

The chosen text-to-animation model was integrated to showcase and prove the algorithm’s core capabilities. For that single purpose, the output was acceptable, but not enough talking about launching the product.

To resolve this problem, we covered the conversion of the generated output into more appropriate formats and created a suitable skeletal framework.

Output categorization

The quality of the generated output was ranging very significantly in dependence of the textual instructions. Some results were undesirable and couldn’t be used for products.

To resolve the problem, we designed and trained a tailored ML model to perform output categorization.

Summing up

Abto Software has proven the feasibility of further software development to discover additional opportunities. Our client has seen great interest in the upcoming product – one month after release, the number of users buying subscriptions has more than doubled.

By bringing the solution to market, our client can leverage:

• New customers, higher demand, and increased revenue streams
• Industry leadership