Accepted to IROS 2025
Tingbang Liang1,2,*, Yixin Zeng1,*, JiaTong Xie1, Boyu Zhou3,†
1 School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China.
2 School of Mechanical Engineering, Xi’an Jiaotong University, Xi’an, China.
3 Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen, China.
*Equal contribution. Listing order is random
†Corresponding Authors
- [30/09/2025]: Release part of the code for DynamicPose.
- [16/06/2025]: DynamicPose is accepted to IROS 2025
[IROS 2025] This repository maintains the implementation of "DynamicPose: Real-time and Robust 6D Object Pose Tracking for Fast-Moving Cameras and Objects".
Please kindly star ⭐ this project if it helps you. Thanks for your support! 💖
Real-world:
real-world-exp.mp4
Pose Tracking Comparison:
real_compare.mp4
conda create -n dynamicpose python=3.9.19
conda activate dynamicpose
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
pip3 install pycryptodomex
pip3 install python-gnupg
- FoundationPose In this step, you should ensure that CUDA_11.8 has been installed.
// install eigen 3.4.0 [optional]
cd $HOME && wget -q https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.tar.gz && \
tar -xzf eigen-3.4.0.tar.gz && \
cd eigen-3.4.0 && mkdir build && cd build
cmake .. -Wno-dev -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_FLAGS=-std=c++14 ..
sudo make install
cd $HOME && rm -rf eigen-3.4.0 eigen-3.4.0.tar.gz
// use conda to install the Python environment
cd ${YOUR_WORKSPACE_PATH}/src/Dynamic_6D/object_pose_estimator/include/FoundationPose
conda activate dynamicpose
python -m pip install -r requirements.txt
python -m pip install --quiet --no-cache-dir git+https://github.com/NVlabs/nvdiffrast.git
python -m pip install --quiet --no-cache-dir kaolin==0.15.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.1.0_cu118.html
python -m pip install "git+https://github.com/facebookresearch/pytorch3d.git"
CMAKE_PREFIX_PATH=$CONDA_PREFIX/lib/python3.9/site-packages/pybind11/share/cmake/pybind11 bash build_all_conda.sh
[1]. Download all network weights from here and put them under the folder weights/
. For the refiner, you will need 2023-10-28-18-33-37
. For scorer, you will need 2024-01-11-20-02-45
.
[2]. Download demo data and extract them under the folder demo_data/
right now, you can check if FoundationPose is ready by:
conda activate dynamicpose
cd ${YOUR_WORKSPACE_PATH}/src/Dynamic_6D/object_pose_estimator/include/FoundationPose
python run_demo.py
- MixFormerV2
cd ${YOUR_WORKSPACE_PATH}/src/Dynamic_6D/object_pose_estimator/include/MixFormerV2
conda activate dynamicpose
pip install tensorboardX jpeg4py lmdb easydict transformers empy
python tracking/create_default_local_file.py --workspace_dir . --data_dir ./data --save_dir .
- Grounding DINO
cd ${YOUR_WORKSPACE_PATH}/src/Dynamic_6D/object_pose_estimator/config/gdino_checkpoints/
conda activate dynamicpose
huggingface-cli download IDEA-Research/grounding-dino-tiny --local-dir grounding-dino-tiny
-
Potential bug fixes [Optional] Using rospy in python=3.9, there are some potential bug:
- version of libffi:
libp11-kit.so.0: undefined symbol: ffi_type_pointer, version LIBFFI_BASE_7.0
BASE_ENV_PATH=$(conda info | grep "base environment" | awk '{print $4}') cd $BASE_ENV_PATH/envs/dynamicpose/lib mv ./libffi.so.7 ./libffi.so.7.bak sudo ln -s /lib/x86_64-linux-gnu/libffi.so.7.1.0 libffi.so.7
-
Download demo bag and extract them under the folder 'demo_bag/'
-
Run Pose Estimator
cd ${YOUR_WORKSPACE_PATH}/ catkin_make roslaunch object_pose_estimator rviz.launch roslaunch object_pose_estimator run_demo.launch
Execution Example:
2025-09-30.22-53-38.mp4
- https://github.com/NVlabs/FoundationPose
- https://github.com/Gy920/segment-anything-2-real-time
- https://github.com/IDEA-Research/GroundingDINO
- https://github.com/MCG-NJU/MixFormerV2
- https://github.com/teal024/FoundationPose-plus-plus
For any technical issues, please contact:
- Yixin Zeng(zengyx53@gmail.com)
- Tingbang Liang(tingbangliang@gmail.com)
@article{liang2025dynamicpose,
title={DynamicPose: Real-time and Robust 6D Object Pose Tracking for Fast-Moving Cameras and Objects},
author={Liang, Tingbang and Zeng, Yixin and Xie, Jiatong and Zhou, Boyu},
journal={arXiv preprint arXiv:2508.11950},
year={2025}
}