Azure OpenAI GPT-4 Turbo with Vision is a large multimodal model (LMM) developed by OpenAI, integrated with Azure’s powerful cloud platform. GPT-4 Turbo with Vision can generate coherent and contextually relevant text based on input prompts, incorporating information from both written text and images. It enables an advanced level of image understanding, allowing for an array of advanced functionalities such as creating elaborate image captions, providing rich contextual descriptions, responding to inquiries about visual content, or assigning intelligent tags.
In this article, I am going to show you how we can consume Azure OpenAI GPT 4 Turbo with Vision API from a Canvas App to analyse an uploaded image.
Prerequisites
- Azure Subscription
- Submit request for Azure Open AI service in your Azure Subscription using this link – https://customervoice.microsoft.com/Pages/ResponsePage.aspx?id=v4j5cvGGr0GRqy180BHbR7en2Ais5pxKtso_Pz4b1_xUNTZBNzRKNlVQSFhZMU9aV09EVzYxWFdORCQlQCN0PWcu
- Power Apps Premium license, as we will be calling a Flow from Power Apps that uses the Premium HTTP connector. You can create a Developer environment to try this out without having the requirement of a Premium license
Create Azure OpenAI Resource
- Navigate to https://portal.azure.com/ , search for Azure Open AI, and click Create
2. Select your Subscription, Select or Create a new Resource Group, provide a Name, and select the Pricing Tier
3. GPT 4 Turbo with vision is currently available in the regions Australia East, Sweden Central, Switzerland North and West US. Select the Region from any of these. In this example I have chosen West US.
4. Continue clicking Next and hit the Create Button.
Create Model Deployments
- Once the Azure Open AI Resource is deployed, go to the resource and click Go to Azure Open AI Studio
2. In the Azure Open AI Studio, from left Navigation, click Deployments >Create New Deployment
3.Select the Model Name GPT-4, Model version vison-preview, provide a name of the model.
Test the model deployment in Chat Playground
- In the Azure AI Studio, from left navigation select Chat.
- In the configuration section, select the GPT 4 vision model you created
- Under Chat Sessions, upload an image and write “Analyse the image” and Send. Here I have used the image of COE Starter Kit Power BI dashboard screenshot, found here https://learn.microsoft.com/en-us/power-platform/guidance/coe/media/pb-5.png
4. It will return the result
Create Power Automate Flow to call the Rest API of Azure OpenAI GPT 4 Turbo with vision
- Create a Power Automate Instant Flow with Power Apps V2 trigger . Add 2 Text input parameters-
- imageContent(Text) – For getting the base 64 of the image uploaded from Power Apps
- prompt (Text) -For getting the prompt
2. Initialize a string variable called result to send back the output to Power Apps
3. Add a HTTP Action
- Method – Post
- URI in the format –
https://{RESOURCE_NAME}.openai.azure.com/openai/deployments/{DEPLOYMENT_NAME}/chat/completions?api-version=2023-12-01-preview where
RESOURCE_NAME is the name of your Azure OpenAI resource
DEPLOYMENT_NAME is the name of your GPT-4 Turbo with Vision model deployment
- API Key – API Key of your resource. You can get it by navigating to your resource in Azure Portal> Keys and EndPoints
Alternatively, you can go to the Chat Playground discussed earlier, click on View Code, retrieve the URI from the EndPoint and the API Key
- Body of the HTTP Action
{
"messages": [
{
"role": "user",
"content": [
@{triggerBody()['text_1']},
{
"image": @{triggerBody()['text']}
}
]
}
],
"max_tokens": 4090,
"stream": false
}
Replace @{triggerBody()[‘text_1’]} with your input parameter for prompt, @{triggerBody()[‘text’]} with your input parameter for imageContent.
Note: This model can have max 4096 tokens
4. Add a Parse JSON action
Content – Body of the HTTP Action
Schema –
{
"type": "object",
"properties": {
"id": {
"type": "string"
},
"object": {
"type": "string"
},
"created": {
"type": "integer"
},
"model": {
"type": "string"
},
"prompt_filter_results": {
"type": "array",
"items": {
"type": "object",
"properties": {
"prompt_index": {
"type": "integer"
},
"content_filter_results": {
"type": "object",
"properties": {
"hate": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
},
"self_harm": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
},
"sexual": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
},
"violence": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
}
}
}
},
"required": [
"prompt_index",
"content_filter_results"
]
}
},
"choices": {
"type": "array",
"items": {
"type": "object",
"properties": {
"finish_details": {
"type": "object",
"properties": {
"type": {
"type": "string"
},
"stop": {
"type": "string"
}
}
},
"index": {
"type": "integer"
},
"message": {
"type": "object",
"properties": {
"role": {
"type": "string"
},
"content": {
"type": "string"
}
}
},
"content_filter_results": {
"type": "object",
"properties": {
"hate": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
},
"self_harm": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
},
"sexual": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
},
"violence": {
"type": "object",
"properties": {
"filtered": {
"type": "boolean"
},
"severity": {
"type": "string"
}
}
}
}
}
},
"required": [
"finish_details",
"index",
"message",
"content_filter_results"
]
}
},
"usage": {
"type": "object",
"properties": {
"prompt_tokens": {
"type": "integer"
},
"completion_tokens": {
"type": "integer"
},
"total_tokens": {
"type": "integer"
}
}
}
}
}
5. Set the variable result with the value content from Parse JSON
6. Return the variable result to Power Apps
Below is the entire Flow-
Canvas App to Upload an Image and Analyse
- Create a Blank Canvas App and add below controls in the screen
- txtPrompt – TextBox to capture prompt
- AddMediaWithImage1 – Add Picture control to upload picture
- lblResult – Label To Show Output
- btnAnalyse – Button to start Analysing
2. Initialize a variable named result at the Onvisible property of the screen with blank value. This will be used to set the value of lblResult Label
3. Set the Text Property of lblResult to this result variable
4. Add the Flow that we created earlier into this Canvas App
4. Add the below code in the OnSelect Property of btnAnalyse
//Set the initial text of the result Label to Please wait
Set(result,"Please wait while we analyze your image....");
//Convert the uploaded image to JSON
Set(imageJSON,JSON(UploadedImage1.Image,JSONFormat.IncludeBinaryData));
//Format the JSON to get rid of from the Left text -"data:image/png;base64,
Set(lefTrimmed,Right(imageJSON,(Len(imageJSON)-Find("base64,",imageJSON)-6)));
//Format the leftTrimmed string to get rid of the right "
Set(formattedBase64,Left(lefTrimmed,(Len(lefTrimmed)-1)));
//Set the result variable to output of Power Automate Flow
Set(result,AnalyzeImageandVideowithGPT4Vision.Run(formattedBase64,txtPrompt.Text).result)
-Here at first we are Converting the image to JSON format and storing in the variable imageJSON . The result will come in the format “data:image/png;base64,<base64 text>“
-We need to extract the base64 text from this string and pass to the Flow for processing .
-We have used Right() function to trim out right chars “data:image/png;base64, and Left function to trim out “
-The base 64 of the image and the prompt is sent to the Flow, and the returned output is displayed in the lblResult label.
Play the App
Upload an image, provide prompt , hit the Analyse Button and get the nicely described detail output of the image provided.
Suparna's Tech Basket 
