"AutoProcessor.from_pretrained" cannot run in distributed environments after v4.56.0

[erikchwang]

Python 3.12

transformers>=4.56.0

Running "AutoProcessor.from_pretrained" (e.g. AutoProcessor.from_pretrained('Qwen/Qwen2.5-VL-7B-Instruct')) in a ray or torchrun environment causes error:

OSError: Can't load image processor for 'Qwen/Qwen2.5-VL-7B-Instruct'. If you were trying to load it from '[https://huggingface.co/models'](https://huggingface.co/models%27), make sure you don't have a local directory with the same name.

This issue happens after v4.56.0.

Downgrading to v4.55.4 can avoid this issue.

[myron]

same issue, looks like a recent bug

[rank7]:     processor = AutoProcessor.from_pretrained(                                                                                                
[rank7]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                
/lib/python3.11/site-packages/transformers/models/auto/processing_auto.py", line 391, 
in from_pretrained                                                                                                                                     
[rank7]:     return processor_class.from_pretrained(                                                                                                   
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                   
/lib/python3.11/site-packages/transformers/processing_utils.py", line 1331, in from_pr
etrained                                                                                                                                               
[rank7]:     args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)                                                        
[rank7]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                        
lib/python3.11/site-packages/transformers/processing_utils.py", line 1390, in _get_ar
guments_from_pretrained                                                                                                                                
[rank7]:     args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))                                                     
[rank7]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                      
lib/python3.11/site-packages/transformers/models/auto/image_processing_auto.py", line
 491, in from_pretrained                                                                                                                               
[rank7]:     raise initial_exception                                                                                                                   
lib/python3.11/site-packages/transformers/models/auto/image_processing_auto.py", line
 473, in from_pretrained                                                                                                                               
[rank7]:     config_dict, _ = ImageProcessingMixin.get_image_processor_dict(                                                                           
[rank7]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                           
lib/python3.11/site-packages/transformers/image_processing_base.py", line 354, in get
_image_processor_dict                                                                                                                                  
[rank7]:     raise OSError(                                                                                                                            
[rank7]: OSError: Can't load image processor for 'Qwen/Qwen2.5-VL-3B-Instruct'. If you were trying to load it from 'https://huggingface.co/models', mak
e sure you don't have a local directory with the same name. Otherwise, make sure 'Qwen/Qwen2.5-VL-3B-Instruct' is the correct path to a directory conta
ining a preprocessor_config.json file

a change to image_processing_base was done recently here 893d89e
(so the issue could be there)

[Rocketknight1]

cc @zucchini-nlp! Also, since distributed env bugs can be hard to track down, it'd be really helpful if someone did a git bisect, or checked if the bug appears in commit 893d89e as we think.

[zucchini-nlp]

@erikchwang hey, can you provide a tiny reproducer since running the following in one machine with 2 processes doesn't produce issues?

# CUDA_VISIBLE_DEVICES=1,2 torchrun --nnodes=1 --nproc-per-node=2  script.py 
from transformers import AutoProcessor

if __name__ == "__main__":
    processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")
    processor

[erikchwang]

I was also running in a similar way, the only difference is that my AutoProcessor.from_pretrained is in a class init method.

[myron]

from transformers import AutoProcessor

if __name__ == "__main__":

    for i in range(10):
      processor = AutoProcessor.from_pretrained(pretrained_model_name_or_path="Qwen/Qwen2.5-VL-3B-Instruct", use_fast=True)
      processor

run on 8 gpus with

accelerate launch --config_file deepspeed_zero2.yaml --num_processes 8 script.py

where deepspeed_zero2.yaml is from here

it doesn't crash every time, but most of the times, seems like some kind of race condition. try running a few times. you should see an error

    resolved_video_processor_file = resolved_video_processor_files[0]                                                                                  
                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^                                                                                  
TypeError: 'NoneType' object is not subscriptable                                                                                                      
                                                                                                                                                       
During handling of the above exception, another exception occurred:                                                                                    
                                                                                                                                                       
Traceback (most recent call last):                                                                                                                     
  File "/lustre/fsw/portfolios/healthcareeng/users/amyronenko/open-r1/test.py", line 7, in <module>                                                    
    processor = AutoProcessor.from_pretrained(pretrained_model_name_or_path="Qwen/Qwen2.5-VL-3B-Instruct", use_fast=True)                              
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                              
  File "python3.11/site-packages/transformers/models/auto/processing_auto.py
", line 391, in from_pretrained                                                                                                                        
    return processor_class.from_pretrained(                                                                                                            
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                            
  File "python3.11/site-packages/transformers/processing_utils.py", line 133
1, in from_pretrained                                                                                                                                  
    args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)                                                                 
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                 
  File "ython3.11/site-packages/transformers/processing_utils.py", line 139
0, in _get_arguments_from_pretrained                                                                                                                   
    args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))                                                              
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                               
  File "python3.11/site-packages/transformers/models/auto/video_processing_a
uto.py", line 305, in from_pretrained                                                                                                                  
    config_dict, _ = BaseVideoProcessor.get_video_processor_dict(pretrained_model_name_or_path, **kwargs)                                              
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                              
  File "/python3.11/site-packages/transformers/video_processing_utils.py", li
ne 707, in get_video_processor_dict                                                                                                                    
    raise OSError(                                                                                                                                     
OSError: Can't load video processor for 'Qwen/Qwen2.5-VL-3B-Instruct'. If you were trying to load it from 'https://huggingface.co/models', make sure yo
u don't have a local directory with the same name. Otherwise, make sure 'Qwen/Qwen2.5-VL-3B-Instruct' is the correct path to a directory containing a v
ideo_preprocessor_config.json file              

[zucchini-nlp]

Thanks for sharing a reproducer, indeed it fails once in a few runs. From visual inspection I see only one difference between 4.55 and 4.56, though it is hard to be sure it is the actual bug

Previously we were downloading one file per processing class which internally calls hf_hub_download directly, but after the release we need to download 2-3 files to support BC with different file names. HF hub internally handles it with snapshot_download utility and calls a thread map for parallelism (here).

We can fix it by calling hf download per file within processor classes, lemme make a few more runs and push a PR

[myron]

@zucchini-nlp in my case the model is already cached locally (under .cache/huggingface/..), so it shouldn't need any downloads, just read from file. but it still crashes as if it doesn't see it locally somehow.

[zucchini-nlp]

The download handles cached files as well and tries to load from cache folder if it exists