c# - How to add image as an input to a ChatMessage with Microsoft.Extensions.AI - Stack Overflow

I'm trying to use Microsoft.Extensions.AI to ask a model hosted in Azure OpenAI to describe an image. I've been able to successfully chat with a gpt-4o-mini model, using the following code:

var message = new ChatMessage(ChatRole.User, "Please describe this image in one sentence");
chatHistory.Add(message);

await foreach (var item in
    chatClient.CompleteStreamingAsync(chatHistory))
{
    Console.Write(item.Text);   
}

However, if I try to add an image to the chat message either locally:

var data = File.ReadAllBytes(@"C:\temp\example.jpg");
message.Contents.Add(new ImageContent(data, "image/jpeg"));

or via a URL:

message.Contents.Add(new ImageContent(new Uri(".jpg")));

Then I get no response from the AI - it just seems to hang for many minutes. I believe that gpt-4o-mini supports image as input. Is what I'm trying to do possible, and if so what is the correct way to add an image to a chat message?

I'm trying to use Microsoft.Extensions.AI to ask a model hosted in Azure OpenAI to describe an image. I've been able to successfully chat with a gpt-4o-mini model, using the following code:

var message = new ChatMessage(ChatRole.User, "Please describe this image in one sentence");
chatHistory.Add(message);

await foreach (var item in
    chatClient.CompleteStreamingAsync(chatHistory))
{
    Console.Write(item.Text);   
}

However, if I try to add an image to the chat message either locally:

var data = File.ReadAllBytes(@"C:\temp\example.jpg");
message.Contents.Add(new ImageContent(data, "image/jpeg"));

or via a URL:

message.Contents.Add(new ImageContent(new Uri("https://example.net/example.jpg")));

Then I get no response from the AI - it just seems to hang for many minutes. I believe that gpt-4o-mini supports image as input. Is what I'm trying to do possible, and if so what is the correct way to add an image to a chat message?

• c#
• artificial-intelligence
• azure-openai
• microsoft.extensions.ai

Share Improve this question Follow asked yesterday Mark HeathMark Heath 49.5k2929 gold badges143143 silver badges200200 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

After trying this in the Chat Playground at ai.azure.com, I found my deployment was being rate-limited. I changed the default of 10K tokens per minute to 40K and the code shown above worked correctly, producing output like:

"A band is performing on stage, featuring instruments like electric guitars, drums, and a keyboard, with a colorful patterned backdrop behind them."

I don't know why the chat client simply hung rather than returning a rate-limit exceeded error.