Process list of list of images

[amyeroberts]

What does this PR do?

Processes a list of list of images e.g. to run the following script:

import requests
from PIL import Image

from transformers import PixtralProcessor, PixtralImageProcessor, AutoTokenizer

url_0 = "https://www.ilankelman.org/stopsigns/australia.jpg"
image_0 = Image.open(requests.get(url_0, stream=True).raw)

url_1 = "http://images.cocodataset.org/val2017/000000039769.jpg"
image_1 = Image.open(requests.get(url_1, stream=True).raw)

url_2 = "https://www.ilankelman.org/stopsigns/australia.jpg"
image_2 = Image.open(requests.get(url_2, stream=True).raw)

image_processor = PixtralImageProcessor()
tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-1.5-7b-hf")
processor = PixtralProcessor(tokenizer=tokenizer, image_processor=image_processor, patch_size=16)

# single image processing
image_inputs = image_0
prompt = "USER: [IMG]\nWhat's the content of the image? ASSISTANT:"
inputs = processor(text=prompt, images=image_inputs, return_tensors="pt", padding=True)


# single list of images
prompt = ["USER: [IMG][IMG]\nWhat's the difference between these two images? ASSISTANT:"]
image_inputs = [image_0, image_1]
inputs = processor(text=prompt, images=image_inputs, return_tensors="pt", padding=True)

# batched list of images
prompt = ["USER: [IMG][IMG]\nWhat's the difference between these two images? ASSISTANT:", "USER: [IMG]\nWhat's the content of the image? ASSISTANT:"]
image_inputs = [[image_0, image_1], [image_2]]
inputs = processor(text=prompt, images=image_inputs, return_tensors="pt", padding=True)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

cc @ArthurZucker

[ArthurZucker]

Thankkkks