Multi-3DLLM Checkpoints

This repository hosts the BeyondSingleObject checkpoints:

• multi-3dllm/: MO3D, Shape Mating, and Change Captioning
• multi-3dllm-classification/: ModelNet40 zero-shot classification

The task checkpoint uses the Patch-Interaction Transformer implementation and was jointly fine-tuned on the three released task datasets. Code and training scripts are available at:

https://github.com/KohsukeIde/BeyondSingleObject

Dataset annotations are available at:

https://huggingface.co/datasets/idekoh/BeyondSingleObject

Point-cloud preparation instructions and upstream download links are provided in the GitHub repository README.

Download

huggingface-cli download idekoh/Multi-3DLLM \
  --local-dir checkpoints \
  --include "multi-3dllm/**" "multi-3dllm-classification/**"

Expected local layout:

checkpoints/
|-- multi-3dllm/
`-- multi-3dllm-classification/
data/

Evaluation Snapshot

Our latest checkpoint and cleaned datasets further improve multi-object 3D reasoning performance.

Task	Metric	Current release
MO3D holistic QA	Binary accuracy B	84.0% (42/50)
Shape Mating	Selection accuracy S	73.0% (146/200)
Change Captioning verify	Binary accuracy B	67.0% (67/100)

The separately released classification checkpoint follows the PointLLM ModelNet40 evaluation protocol. See the repository README for its six one-, two-, and three-object commands.

Usage

Example inference:

MODEL_PATH=checkpoints/multi-3dllm \
ANNO_PATH=data/mo3d/test.json \
DATA_PATH=data/point_clouds \
OUTPUT_DIR=outputs/mo3d_eval \
scripts/eval/infer.sh

ModelNet40 classification:

MODEL_PATH=checkpoints/multi-3dllm-classification \
OUTPUT_DIR=outputs/modelnet40_eval \
LIMIT=0 \
PROMPT_MODE=paper \
NUM_OBJECTS=1 \
TARGET_POSITION=1 \
scripts/eval/eval_modelnet.sh

Notes

LLM-judged reasoning and delta-caption scores depend on the judge model and prompt configuration. Use the released evaluators and report the exact judge configuration when comparing those metrics. The task and classification checkpoints have different roles and should not be interchanged.

License

These checkpoints are built with the BeyondSingleObject codebase and use PointLLM-style initialization and data. They may inherit terms from upstream model, code, and dataset components, including PointLLM, Vicuna/Llama, Objaverse/Cap3D, ShapeTalk, Thingi10K, Neural Shape Mating, and ModelNet40. Please check the corresponding upstream licenses before redistribution or commercial use.

Citation

@inproceedings{ide2026beyondsingleobject,
  title={BeyondSingleObject: Learning 3D Relations with Large Language Models},
  author={Ide, Kohsuke and Yamada, Ryousuke and Qiu, Yue and Ma, Xianzheng and Fukuhara, Yoshihiro and Kataoka, Hirokatsu and Satoh, Yutaka},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Findings},
  year={2026}
}