Category-Level Shape and Pose Estimation in Less Than a Millisecond

[Paper] [Video]

Official Julia implementation of "Category-Level Object Shape and Pose Estimation in Less Than a Millisecond" by Lorenzo Shaikewitz, Tim Nguyen, and Luca Carlone

One Run	Multiple Minima

This repository contains the solver. We open-source keypoint detection and training here.

Quick Start

First, make sure you have Julia installed. This repository was tested with v1.11.6. Then, clone the repository and follow the directions below. We assume you are in the repo folder.

1. Clone this repository

git clone https://github.com/lopenguin/Fast-ShapeAndPose.git
cd Fast-ShapeAndPose

2. Open the Julia REPL

julia --project

3. Install dependencies

using Pkg
Pkg.add(url="https://github.com/lopenguin/SimpleRotations.jl")

That's it! I've prepared a simple demo script which measures runtime and plots the SCF iterates:

julia> include("scripts/demo_pace.jl")

The script may take a bit to run the first time while Julia precompiles it, but after that it should be much faster!

Reproducing Results

First, download the data for each experiment:

wget -O data.zip "https://drive.usercontent.google.com/download?id=1Z229oE_R27QneU0_GCJS2oH3yznJ8KdF&export=download&confirm=yes"
unzip data.zip -d data
rm data.zip

You can also download from Google drive and put it in a "data" folder.

Now, you can print a summary of any of the experiments by calling:

julia --project synthetic.jl
# cast.jl, nocs.jl, apollo2.jl

To re-run a specific experiment, add the method name and --force. For example:

julia --project synthetic.jl SCF --force

Running multiple methods together allows Julia to precompile for the specific problem data, which artificially increases the runtime of the second or third method.

Below, we provide more details on each of the experiments.

Synthetic Experiments

The synthetic experiment data is all in this file. All methods have the same initial condition. The problem data is generated using:

q0 = normalize(randn(4))
serialize("data/synthetic/problem_data.dat", (data,q0))

CAST Experiments

We test on the CAST dataset. First, we run the fast compatibility checks ROBIN using this python script. This saves the inlier clique to the "robin_" json file along with the time it takes to run ROBIN. For visualization, see scripts/visualization/cast.jl. The keypoint detector is not released.

NOCS Experiments

We test on the NOCS-REAL275 dataset. Keypoint detection is done separately. Save files are available here and scripts to run keypoint detection are available here. We caution that keypoint detection doesn't work particularly well.

To run on NOCS, you also need to specify the subfolder (see data/nocs). For example:

julia --project nocs.jl SCF mug --force

There are also several visualization scripts in the visualization folder.

ApolloCar3D Experiments

Unfortunately, most ApolloCar3D data is no longer available at their official repository. We use the code in this repository to process the dataset. While we don't have access to the original images, the keypoint detections are provided in the data.

BibTeX

@misc{Shaikewitz25arxiv-FastShapeAndPose,
      title={Category-Level Object Shape and Pose Estimation in Less Than a Millisecond}, 
      author={Lorenzo Shaikewitz and Tim Nguyen and Luca Carlone},
      year={2025},
      eprint={2509.18979},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2509.18979}, 
}