RuntimeError: CUDA error: an illegal memory access was encountered

[SunHongyang10]

Hello~ Wonderful Work!

I am trying to run the train_coarse.py, then I meet an error:

I tried to solve it, but I failed😭

Is there a problem with my virtual environment?

[Vilour]

Same error here. Have you solved it?

[Snosixtyboo]

Hi,

unfortunately, this error is not super specific, we have seen it before in 3D Gaussian Splatting. We tried our best to replicate it, but we were never able to get it on any of our machines, so we never worked out how to debug it...

Could you let us know your OS / GPU (how many GPUs are in your machine)? Getting the latest NVIDIA drivers might help, but bottom line, without full access to a setup where it happens, it might be really tough to find it.

[Vilour]

Hi,

unfortunately, this error is not super specific, we have seen it before in 3D Gaussian Splatting. We tried our best to replicate it, but we were never able to get it on any of our machines, so we never worked out how to debug it...

Could you let us know your OS / GPU (how many GPUs are in your machine)? Getting the latest NVIDIA drivers might help, but bottom line, without full access to a setup where it happens, it might be really tough to find it.

Hi,

I'm working with Ubuntu 18.04 with one GPU (RTX 3090). I used this setup in 3D Gaussian Splatting before and it works fine. By the way, my nvcc -V returns 11.6 and I created the virtual environment as the repo describes. Does it associate with CUDA version? Is visibility_filter really working here? Maybe I can just comment this line..

[Vilour]

The error persisted even if I comment this line..

[ameuleman]

Hi,
Did you install pytorch corresponding to cuda 11.x?
pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu118

[Vilour]

Hi, Did you install pytorch corresponding to cuda 11.x? pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --index-url https://download.pytorch.org/whl/cu118

Yes, I installed pytorch with this command. By the way, the error on my machine appeared at 10040 iterations, not from the beginning.

[Vilour]

Hi,

I tied on another machine (Ubuntu 20.04/A6000) with the same dataset, and the error appears again on the 10040 iterations. I guess this issue associates with dataset?

[ameuleman]

I just tried downloading SmallCity and running full_train.py. The coarse optimization went smoothly. Are we working with the same dataset?

[Vilour]

I just tried downloading SmallCity and running full_train.py. The coarse optimization went smoothly. Are we working with the same dataset?

I was working with a dataset which I collected myself. I'm trying with SmallCity right now.

[Vilour]

I just tried downloading SmallCity and running full_train.py. The coarse optimization went smoothly. Are we working with the same dataset?

I just tried with SmallCity and the error appeared immediately.

[Snosixtyboo]

@Vilour @SunHongyang10 When it fails, could you try keep an eye on the GPU memory consumption? Is it possible that the system goes out of video memory? This should not happen on a 3090...

[SunHongyang10]

@Vilour @SunHongyang10 When it fails, could you try keep an eye on the GPU memory consumption? Is it possible that the system goes out of video memory? This should not happen on a 3090...

I just tried with small_city dataset, and it fails immediately, my device is a 3090

[anchun]

Same here in Ubuntu20.04, with the following call stacks:

File "train_coarse.py", line 190, in
training(lp.extract(args), op.extract(args), pp.extract(args), args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
File "train_coarse.py", line 106, in training
loss.backward()
File "/home/anchun/software/miniconda3/envs/hierarchical_3d_gaussians/lib/python3.12/site-packages/torch/_tensor.py", line 525, in backward
torch.autograd.backward(
File "/home/anchun/software/miniconda3/envs/hierarchical_3d_gaussians/lib/python3.12/site-packages/torch/autograd/init.py", line 267, in backward
_engine_run_backward(
File "/home/anchun/software/miniconda3/envs/hierarchical_3d_gaussians/lib/python3.12/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered

[Vilour]

@Vilour @SunHongyang10 When it fails, could you try keep an eye on the GPU memory consumption? Is it possible that the system goes out of video memory? This should not happen on a 3090...

The video memory consumption is ok. Only takes a few gigabytes.

[ameuleman]

Hi,

Could you please provide cuda and nvidia driver versions?