12

JSON Parsing using GPT-3.5

An early experiment to parse JSON using GPT 3.5 demonstrating its accuracy in converting visual and text context to structured json outputs which directly can be used as intelligence plug-in for apps.

json parsingaigpt-3.5

The Problem: Lost Receipts and Uncontrolled Expenses

We've all been there. That mini heartbreak after losing a grocery receipt and realizing we've lost track of our expenses. "You must gain control over your money or the lack of it will forever control you," Dave Ramsey once said. And I felt that loss of control every time I couldn't account for every penny spent.

For those who love precision like I do, this is a familiar pain. Two weeks back, I highlighted how easily I missed tracking individual expenses during shopping trips. Imagine heading to Walmart and trying to recall the price of the tomatoes or the exact cost of the meat and dairy you purchased last time. Frustrating, isn't it?

So, I embarked on a journey to find a solution to this ubiquitous problem.

The Solution: OCR + GPT-3.5 Turbo

Discovering the Right Tools

I stumbled upon a remarkable use of ChatGPT: extracting structured data from an OCR response of unstructured raw material. While there are existing OCR apps like Synthetiq.co which harness the power of OpenAI to outperform even giants like Google Document AI, the costs can be prohibitive for individual use.

This led me to explore a solution that combines both open-source and commercial resources to find the best fit for my needs.

Why Mindee DocTR?

After days of trial and error with various OCR tools like Google Vision, Document AI, and Tesseract, I discovered the gem that is Mindee DocTR. Powered by TensorFlow 2 & PyTorch, DocTR offers a seamless way to parse textual information from documents.

Key Advantages of DocTR:

  • Optimized for document analysis, particularly for printed documents such as invoices and receipts
  • Excels in recognizing clear, printed text from high-resolution scans or photographs
  • Proficient in scanning printed receipts with good contrast between text and background
  • Adept at processing structured documents like receipts that follow consistent formats
  • Handles screenshot receipts with remarkable accuracy when quality is good
  • Performance can be boosted with proper preprocessing like alignment and enhancement

While Mindee's DocTR proved to be a perfect fit for my needs, always consider evaluating multiple OCR tools based on your specific requirements. Different tasks or languages might have an OCR solution better tailored to them.

The Technical Implementation

System Architecture

The solution built using DocTR consists of three main components:

  1. OCR Processing: Leveraging Flask and DocTR, I built a system where images of receipts uploaded are processed in real time. The textual content is meticulously extracted and organized into blocks, lines, and words.

  2. Text Structuring: Recognizing the unique formats of receipts, I ensured lines of text that are vertically close (like item names and prices) are intelligently merged. This ensures each item and its cost is paired correctly.

  3. API Endpoints: A dedicated endpoint for processing each image was set up, streamlining the entire operation.

Example: Text Extraction Process

Here's how the magic works. When a receipt is processed through DocTR, it produces raw text output like this:

srveywalmart.com
SEETET
FR our, Walmart *
863-299-5527 Mgr:RAYMOND
355 CYPRESS GARDENS BLVD
WINTER HAVEN FL. 33880
00968 OP# 009038 TE# 38 TR# 06610
STH CCKTLS 004450005190 F 5.98 N
HF2 007874235186 F 3.54 0
GV WHOLE WHIP TOP 007874237173 F 1.72 N
REG CRM CHSE 2PK 007874203270 F 2.56 N
GRD BISCUIT 001800000188 F 1.24 N
GV FF ONIONS 007874210132 F 2.28 N
GV PWD 2LB 007874237219 F 1.62 N
GV VAN PUD 007874206280 F 0.84 N
GV VAN PUD 007874206280 F 0.84 N
CRM OF MSHRM 005100001261 F 0.98 N
GV COND MILK 007874243304 F 1.12 N
GV COND MILK 007874243304 F 1.12 N
GV WHT BKG 007874200059 F 1.88 N
GV WHT BKG 007874200039 F 1.88 N
GV PEC CHOP 007874220128 F 9.48 N
GV TW SH CH 007874208495 F 1.88 N
GV BRD ROUND 007874228544 F 0.84 N
JEGGING 088496824373 12.96 X
LEGGING 880979376907 5.96 X
SUBTOTAL 58.72
TAX 1 7.000 % 1.32
TOTAL 60.04
DEBIT TEND 60.04
CHANGE DUE 0.00
EFT DEBIT PAY FROM PRIMARY
60.04 TOTAL PURCHASE
FIFTHTHIRD DEBIT ** **** **** 9832I0
REF # 131500139873
NETWORK ID. 0076 APPR CODE 713070
US Debit
AID A0000000042203
AAC 1D4415A920011FE4
TERMINAL # SC010076
11/11/21 09:42:12
#I ITEMS SOLD 19
TC# 55..937612.10
Y40
0050
11/11/21 09:42:17

GPT-3.5 Turbo Transformation

This raw output is then fed into GPT-3.5 Turbo to structure it into clean, usable JSON:

{
  "store_information": {
    "name": "Walmart",
    "address": "355 CYPRESS GARDENS BLVD, WINTER HAVEN FL. 33880",
    "phone": "863-299-5527",
    "manager": "RAYMOND",
    "store_website": "srveywalmart.com"
  },
  "transaction_details": {
    "operator": "00968",
    "terminal": "38",
    "transaction": "06610"
  },
  "items": [
    {"description": "STH CCKTLS", "code": "004450005190", "price": "5.98"},
    {"description": "HF2", "code": "007874235186", "price": "3.54"},
    {"description": "GV WHOLE WHIP TOP", "code": "007874237173", "price": "1.72"},
    {"description": "REG CRM CHSE 2PK", "code": "007874203270", "price": "2.56"},
    {"description": "GRD BISCUIT", "code": "001800000188", "price": "1.24"},
    {"description": "GV FF ONIONS", "code": "007874210132", "price": "2.28"},
    {"description": "GV PWD 2LB", "code": "007874237219", "price": "1.62"},
    {"description": "GV VAN PUD", "code": "007874206280", "price": "0.84"},
    {"description": "GV VAN PUD", "code": "007874206280", "price": "0.84"},
    {"description": "CRM OF MSHRM", "code": "005100001261", "price": "0.98"},
    {"description": "GV COND MILK", "code": "007874243304", "price": "1.12"},
    {"description": "GV COND MILK", "code": "007874243304", "price": "1.12"},
    {"description": "GV WHT BKG", "code": "007874200059", "price": "1.88"},
    {"description": "GV WHT BKG", "code": "007874200039", "price": "1.88"},
    {"description": "GV PEC CHOP", "code": "007874220128", "price": "9.48"},
    {"description": "GV TW SH CH", "code": "007874208495", "price": "1.88"},
    {"description": "GV BRD ROUND", "code": "007874228544", "price": "0.84"},
    {"description": "JEGGING", "code": "088496824373", "price": "12.96"},
    {"description": "LEGGING", "code": "880979376907", "price": "5.96"}
  ],
  "summary": {
    "subtotal": "58.72",
    "tax_percentage": "7.000%",
    "tax_amount": "1.32",
    "total": "60.04",
    "payment_method": "DEBIT TEND",
    "payment_amount": "60.04",
    "change_due": "0.00"
  },
  "payment_details": {
    "debit_source": "FIFTHTHIRD DEBIT",
    "last_four_digits": "9832I0",
    "reference_number": "131500139873",
    "network_id": "0076",
    "approval_code": "713070",
    "terminal_number": "SC010076"
  },
  "timestamp": "11/11/21 09:42:12",
  "items_sold_count": 19,
  "transaction_code": "55..937612.10"
}

Integration with Personal Finance Management

With this system in place, my personal finances became a lot brighter. Now, whether it's a grocery receipt, purchase order, insurance receipt, or any document, I can swiftly extract and save its details, right down to the tax.

Key Benefits

Simplified Data Entry

In our fast-paced world, every saved minute counts. Traditional methods of keeping track of expenses involved manual entries, pouring over receipts, and the inevitable human errors. However, the newly developed system eradicates these inconveniences:

  • Automatic Scanning: Instead of manually inputting each item and its associated cost, users simply upload their transaction document. The system takes over from there, scanning every line with precision.

  • Comprehensive Data Capture: Whether it's the tax on a purchase, a discount applied, or individual line items, the system meticulously captures each detail. No more trying to remember or manually calculate the sales tax on a purchase.

  • Accessible Archive: Each processed document is stored within the system, creating a digital archive of all transactions. Lost your physical receipt? No worries. The digital copy ensures you always have access to past transactions.

Intelligent Categorization

The magic doesn't stop at data extraction. The platform's intelligent categorization system ensures each transaction is sorted, organized, and easily accessible:

  • Item Recognition: Bought tomatoes? The system identifies individual items, even if they are as commonplace as a tomato. This level of granularity ensures that users have a clear picture of where their money is going, down to the last penny.

  • Contextual Understanding: Beyond just recognizing items, the system understands the context. For instance, if you purchase tomatoes at a grocery store, it's categorized under 'Groceries'. But if it's from a nursery, it might be categorized under 'Gardening' or 'Home Improvement'.

  • Automated Sorting: No more manual tagging or sorting. The system automatically classifies each item under predefined categories, be it groceries, dining, entertainment, utilities, or any other segment.

  • Custom Categories: While the system offers a comprehensive set of predefined categories, users have the flexibility to create custom categories. This personalization ensures that the system adapts to individual needs, rather than the other way around.

Future Potential and Advanced Features

While the above are the basic features of the system, more features could be built over the granular data that needs to be managed. Given the dynamic nature of the financial world, and the increasing need for efficient money management, the integration of advanced OCR technology and ChatGPT's AI capabilities presents limitless opportunities:

Predictive Analysis

With access to historical data from receipts, it's feasible to develop a predictive model to forecast future spending. This will allow users to plan their finances better, anticipate expenses, and even receive alerts for probable overspending based on past behavior.

Advanced Budgeting

Leveraging AI, users can set a budget for different categories. The system would then provide real-time feedback on spending against that budget, offering suggestions or warnings as necessary.

Expense Sharing

Think of situations where you might be sharing expenses with someone - roommates, group trips, or shared office expenses. The system could facilitate automatic splitting and tracking of shared costs, streamlining the process.

Conclusion

The integration of OCR technology like DocTR and AI capabilities of ChatGPT to manage personal finances is just scratching the surface of what's possible. The true potential lies in the continuous evolution and integration of these technologies to make financial management more efficient, accurate, and user-friendly.

Building a personal finance management solution with these features is not just about managing money; it's about regaining control over our financial lives. I will be writing more about the App System UX integrated with these features and I'm excited about the possibilities and hope to share more insights, developments, and breakthroughs with all of you in the upcoming days.

So, follow along and stay tuned for more updates. Until then, happy budgeting and wise spending!