Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vision Support in OpenaiChat Provider #2461

Open
The2Alert opened this issue Dec 6, 2024 · 2 comments
Open

Vision Support in OpenaiChat Provider #2461

The2Alert opened this issue Dec 6, 2024 · 2 comments

Comments

@The2Alert
Copy link

The2Alert commented Dec 6, 2024

Request

Please improve the Vision function so that you can transfer several pre-loaded images.
Image parameters are not processed.

API HTTP Request
/v1/chat/completions

Example Body

{
    "model": "gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What are in these images? Is there any difference between them?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
            }
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }

OpenAI Official Docs
https://platform.openai.com/docs/guides/vision

Only text :(

"content": {"content_type": "text", "parts": [message["content"]]},

@hlohaus
Copy link
Collaborator

hlohaus commented Dec 8, 2024

Image data URIs are accepted within the designated image field; embedding them within message content is not supported. Currently, only a single image is accommodated due to ongoing efforts to ensure broad provider compatibility.

@The2Alert
Copy link
Author

add support for multiple images
Screenshot 2024-12-10 210035
Screenshot 2024-12-10 210056

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants