Extracting Information from Unstructured Documents at Scale with Amazon Bedrock Data Automation

Extracting valuable insights from unstructured documents has become a pressing need for businesses across various industries. Many organizations grapple with tasks like creating product feature tables from vague descriptions, extracting metadata from documents, and analyzing legal contracts, customer reviews, or news articles. In tackling these challenges, Named Entity Recognition (NER) traditionally stood at the forefront. NER is designed to identify entities within predefined categories such as people and organizations. However, this approach has limitations—it typically focuses on text documents and is confined to a fixed entity set, making it inadequate for comprehensive analysis of diverse data types, such as numeric scores or varied text formats.

The New Paradigm: Generative AI

The advent of Generative AI has transformed this space, unlocking capabilities that extend far beyond conventional methods. By enabling intelligent document processing (IDP) without the need for costly data annotation or complex model training, Generative AI empowers organizations to extract information more efficiently and accurately.

The Launch of Amazon Bedrock Data Automation

In this context, AWS recently announced the general availability of Amazon Bedrock Data Automation. This feature of Amazon Bedrock automates the generation of insights from unstructured, multimodal content, including documents, images, videos, and audio. The service’s pre-built capabilities simplify IDP and information extraction through a unified API, eliminating the necessity for intricate prompt engineering. This positions Amazon Bedrock Data Automation as an excellent choice for organizations looking to streamline their document processing workflows at scale.

Key Advantages of Amazon Bedrock Data Automation

Amazon Bedrock Data Automation stands out for its simplicity, industry-leading accuracy, and managed service capabilities. It autonomously handles complexities like document parsing, context management, and model selection, allowing developers to focus on their core business logic rather than get bogged down by the intricacies of IDP implementation.

While this service satisfies most IDP needs, some organizations may require additional customization. For instance, regulatory requirements might dictate that certain organizations utilize self-hosted foundation models (FMs). Additionally, companies often prefer to maintain full control over the IDP pipeline rather than relying on a managed service, especially in regions where Amazon Bedrock Data Automation isn’t yet available, such as beyond us-west-2 and us-east-1.

A Comprehensive IDP Application

To assist those interested in deploying this solution, AWS provides an end-to-end IDP application powered by Amazon Bedrock Data Automation and other AWS services. This application offers reusable infrastructure as code (IaC) to establish an IDP pipeline, along with an intuitive UI. Users only need to submit input documents—such as contracts or emails—and list the attributes they wish to extract. The application then leverages generative AI to perform IDP seamlessly.

Solution Overview

The IDP solution is deployed as IaC through the AWS Cloud Development Kit (AWS CDK), with Amazon Bedrock Data Automation serving as the operational engine for information extraction. For scenarios that necessitate further customization, alternative processing options involving Amazon Bedrock FMs and Amazon Textract can be integrated.

The orchestration of the IDP workflow is executed using AWS Step Functions, which allow for the parallel processing of multiple documents—and components like AWS Lambda functions play critical roles in calling Amazon Bedrock Data Automation or Amazon Textract depending on the user’s selected parsing mode. Processed documents and extracted data points are stored in Amazon S3.

Workflow Steps

Here’s a breakdown of the IDP workflow:

1. Users log in to the web application via Amazon Cognito, choose input documents, and outline the fields they want to extract.
2. Upon initiating the IDP pipeline, the application generates a pre-signed S3 URL for document uploads to Amazon S3.
3. The application triggers Step Functions to activate the workflow, processing documents based on user-defined inputs.
4. Using specific Lambda functions, documents are directed to either Amazon Bedrock Data Automation or Textract, depending on their type and selected parsing mode.
5. For organizations opting for Amazon Bedrock FMs, images of documents are sent to a multimodal FM for content extraction.
6. Finally, the application checks the state machine execution results periodically and relays extracted attributes back to users.

Prerequisites for Deployment

Deployment of this solution can occur from either a local computer or an Amazon SageMaker notebook instance. Users must have access to an AWS account with the necessary permissions to create and launch a SageMaker notebook instance.

Step-by-Step Deployment

To deploy the IDP solution:

1. Navigate to the AWS Management Console and choose your desired Region.
2. Launch a SageMaker notebook instance.
3. Clone the solution repository from GitHub.
4. Install required dependencies and set up your environment.
5. Modify the configuration file for stack names and other parameters.
6. Finally, bootstrap and deploy the AWS CDK in your account, which may take some time to complete.

Real-World Applications of IDP

Analyzing Financial Documents

One key application of the IDP solution is in extracting metrics from multi-page financial statements. Users can upload a sample document and specify the metrics they want to extract—such as current assets or operating profit. The system leverages Amazon Bedrock Data Automation to parse the documents and extract relevant numerical data efficiently. The results are then presented in a table format, making it easy for users to analyze the insights derived from complex financial documents.

Processing Customer Emails

Another practical use case involves customer complaints received via email. Users can upload customer emails and specify various attributes to extract, from customer names to shipment delays. The application’s capability to handle varied data types ensures that businesses can efficiently process and analyze customer sentiment and issues without manual intervention.

Pricing Estimates

Understanding the cost implications of deploying IDP solutions is essential. Amazon Bedrock Data Automation has a pricing model based on the size of the input documents, while additional costs may accrue when using Amazon Textract for OCR or when invoking Amazon Bedrock FMs.

Approximate pricing for using these services can vary. For instance, processing 100 20-page financial documents may cost $20 with Amazon Bedrock Data Automation, while processing 100 one-page emails could be around $1. If organizations use a mix of Amazon Textract and Bedrock FMs, costs may also fluctuate based on specific processing tasks.

Future Enhancements and Considerations

As advancements continue in the realm of Generative AI, AWS plans to expand support for state-of-the-art language models available through Amazon Bedrock. Future iterations of the IDP solution aim to enhance document processing capabilities, enabling better management of larger documents without fragmenting them.

Organizations looking to capitalize on this powerful tool can find a wealth of information in the GitHub repository and resources dedicated to Amazon Bedrock. This is an exciting era for businesses seeking to leverage Generative AI, making document processing more accessible and effective than ever before.