DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation

Guanxiong Chen1,3,*,†, Yiduo Qu2,*, Qianjun Xia1,*, Pengyu Jing3, Yixian Cheng1, Bole Ma4, Pengzhi Yang3, Bingyang Zhou3, Ziming Li3, Shashwat Suri1, Gongbo Sun5, Chao Liu1, Peter Yichen Chen1, Ziqiu Zeng3,+, Fan Shi3,+
* Equal technical contribution. † Project lead. + Corresponding author.
1 University of British Columbia, Vancouver, Canada 2 University of Cambridge, Cambridge, United Kingdom 3 National University of Singapore, Singapore 4 NHR@FAU, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany 5 University of Wisconsin–Madison, Madison, WI, USA
Paper Arxiv Code

Overview

From a single image to a deformable asset that can be tested, repaired and used in simulation.

Our contributions

  1. From one image to a deformable asset

    DiagGen generates part-aware geometry and material properties from an everyday image, then tests the resulting asset through controlled interactions in a physics simulator.

  2. Simulation feedback guides targeted repair

    A diagnostic agent probes informative regions and sends repair cues to segmentation, material inference or mesh processing. Evaluation on 40 assets shows useful feedback and moderate improvements after repair.

  3. Assets for robotic simulation

    Generated assets support contact-rich manipulation and integration into reconstructed real-world scenes. Their connectivity and material behavior make a visible difference in downstream use.

Asset gallery

Explore the geometry, part structure and materials of generated everyday objects.

Interactive 3D

Explore the assets

Rotate six DiagGen assets and switch between textured surfaces, tetrahedral boundaries, source detail and part segmentation.

Dragon

A closer look at the generated asset.

3D preview
Your browser does not support the canvas needed for this interactive 3D preview.
Display mode

The 3D viewer loads as you approach.

Drag to rotateScroll to zoomRight-drag to pan

More generated assets

From the input photograph to a generated object, with a closer look at its different material regions.

Each asset begins with a single phone or internet image. The gallery shows generated meshes and their part-wise material specifications.

Method

Generate. Simulate. Diagnose. Refine. Physical interactions provide feedback for the next revision.

01 / Generate

Generation pipeline

DiagGen converts a single image into a deformable asset through a generate–simulate–diagnose–refine loop. Stage agents coordinate geometry, material inference and meshing; model backends support their reasoning, and tools execute and validate each stage. Simulation evidence can route a candidate back for repair before export.

View the architecture diagram
DiagGen architecture: stage agents direct model backends and tools through image cleanup, segmentation, part reconstruction, material inference, mesh processing and simulator diagnostics; repair routes return to upstream stages.
Architecture of DiagGen. Select the figure to open it at full resolution.

02 / Test and diagnose

Simulation-based diagnostics

The agent selects a meaningful region, applies a controlled grasp, and observes deformation and recovery after release. It can run another probe when the evidence is insufficient, then recommend acceptance or a repair at the responsible generation stage.

View the diagnostic workflow
Diagnostic agent workflow: plan an interaction, target an anchor and grasp region, simulate compression, observe maximum deformation and release, and assess whether to accept or repair.
Diagnostic-agent workflow, including the anchor, grasp region, deformation and recovery observations.

03 / Repair and evaluate

What repair changes

A USB hub before and after diagnostics-guided segmentation repair, under the same grasp-and-lift action.

Before repair
After repair
0:00

Before repair, the cable retains an overly stiff bend. After repair, the cable is represented separately from the case, allowing it to stay connected, deform under gravity and let the connector hang naturally.

Across the revised assets

Better part–material specifications

+1.29 points

Mean PMSC improves from 2.14 to 3.43 on the five-point scale.

Improved
10
Unchanged
4
Declined
0

Paired results for the 14 assets selected for repair out of 40 evaluated. PMSC measures consistency of the static part and material specification with the source image and material priors. Read the evaluation in the paper ↗

Simulation applications

Generated deformable objects in contact-rich robotic tasks.

Dino · Bimanual twisting

11 s · 1×

Two grippers twist the soft dinosaur. Watch how the body deforms while staying connected.

Only the manipulated deformable object is generated by our pipeline. Tasks are dynamically simulated in Genesis.

Real-world experiments

Real robot manipulations and their physics-based simulation counterparts.

Plush · Pick and place

11.4 s · 1×

An AIRBOT places a plush toy in a basket. The simulation renders the robot, plush and reconstructed basket over a real background.

Full video

The complete DiagGen project film, from motivation to applications.

3 min · Sound available in the playerWatch on YouTube ↗

Citation

If you find this work useful, please cite DiagGen.

BibTeX
@misc{chen2026diaggenagenticgenerationdeformable,
  title        = {DiagGen: Agentic Generation of Deformable Assets with Sim-based Diagnostics for Robotic Simulation},
  author       = {Guanxiong Chen and Yiduo Qu and Qianjun Xia and Pengyu Jing and Yixian Cheng and Bole Ma and Pengzhi Yang and Bingyang Zhou and Ziming Li and Shashwat Suri and Gongbo Sun and Chao Liu and Peter Yichen Chen and Ziqiu Zeng and Fan Shi},
  year         = {2026},
  eprint       = {2609.23103},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url          = {https://arxiv.org/abs/2609.23103},
}