Skip to content

Commit 098ceb2

Browse files
author
Dinesh Sajwan
committed
feat(documentation): updated qa readme for image qa
1 parent 16ac265 commit 098ceb2

File tree

1 file changed

+6
-3
lines changed
  • src/patterns/gen-ai/aws-qa-appsync-opensearch

1 file changed

+6
-3
lines changed

src/patterns/gen-ai/aws-qa-appsync-opensearch/README.md

+6-3
Original file line numberDiff line numberDiff line change
@@ -42,10 +42,13 @@
4242

4343
This construct provides a question answering workflow (RAG + long context window) using Amazon Bedrock and a provisioned Amazon OpenSearch cluster.
4444

45-
- If a document is provided as an input to the AppSync query, the AWS Lambda function will first verify the length of the document. If the document size is above the max number of tokens for the selected model, the Lambda will query the knowledge base (similarity search) and filter by document name. This assumes that the chunks of texts stored in the knowledge base have the document name as metadata. Otherwise, the content of the document is provided to the LLM as part of the context.
45+
- If a pdf document is provided as an input to the AppSync query, the AWS Lambda function will first verify the length of the document. If the document size is above the max number of tokens for the selected model, the Lambda will query the knowledge base (similarity search) and filter by document name. This assumes that the chunks of texts stored in the knowledge base have the document name as metadata. Otherwise, the content of the document is provided to the LLM as part of the context.
4646
- If no document is provided as input, the Lambda will perform a similarity search against the entire knowledge base.
47+
- Utilizing AppSync queries, images can be provided as inputs to invoke AWS Lambda functions that leverage Amazon SageMaker-deployed Idefics multimodal for visual question answering. The Idefics model from Hugging Face enables both retrieval-augmented generation (RAG) and long context strategies when processing images for visual QA. For details on deploying Idefics modal from Hugging Face to SageMaker, please refer to the "AWS Model Deployment on SageMaker" guide using Hugging Face models.
48+
- If an image file (.jpg,.jpeg,.png) is provided as an input to the AppSync query, the AWS Lambda function leverage Amazon SageMaker-deployed Idefics multimodal for visual question answering. The construct support both RAG to find a relevant image from the knowledge base and LONG_CONTEXT to answer the questions on a single image. To deploy Idefics modal from Hugging face on sagemaker please refer [aws-model-deployment-sagemake](../aws-model-deployment-sagemaker/README_hugging_face.md).
4749

48-
The construct uses Amazon Bedrock as the large language model provider. amazon.titan-embed-text-v1 is used as the embeddings model to query the knowledge base (provisioned Amazon OpenSearch cluster), and anthropic.claude-v2:1 for question answering. Make sure both models are enabled in your account. Please follow the [Amazon Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for steps related to enabling model access.
50+
51+
The construct uses Amazon Bedrock as the large language model provider. amazon.titan-embed-text-v1 is used as the embeddings model for text and amazon.titan-embed-image-v1 for images to query the knowledge base (provisioned Amazon OpenSearch cluster), and anthropic.claude-v2:1 for question answering. Make sure both models are enabled in your account. Please follow the [Amazon Bedrock User Guide](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html) for steps related to enabling model access.
4952

5053
The input document must be stored in the input Amazon Simple Storage Service bucket in text format (.txt). Another construct is available to ingest and process files to text format and store them in a knowledge base: [aws-rag-appsync-stepfn-opensearch](../aws-rag-appsync-stepfn-opensearch/README.md).
5154

@@ -102,7 +105,7 @@ os_domain = os.Domain.from_domain_attributes(
102105
)
103106

104107
# get an existing userpool
105-
cognito_pool_id = 'us-east-1_XXXXX';
108+
cognito_pool_id = 'us-east-1_XXXXX'
106109
user_pool_loaded = cognito.UserPool.from_user_pool_id(
107110
self,
108111
'myuserpool',

0 commit comments

Comments
 (0)