I Tried to Automate My Taxes with AI.

The Problem
It’s already mid-April and once again I’m late to write my 確定申告 (income tax return).
I do want to pay the minimum legal amount - who doesn’t?!
But the idea of going over receipts and categorizing expenses feels as appealing as a colonoscopy.
So I Asked ChatGPT to Do It
So of course, I end up doing what any modern, broken person does:
I ask ChatGPT, “How can AI do my taxes for me?”It thinks for a moment and replies: “To be free, you have to own the boring parts too.”
I forget I’ve told it to speak like a mix of Marcus Aurelius and a Zen monk — but this time?
“Go fuck yourself, robot Yoda.”
Then I Lost the Bag
I sit down to start my 確定申告. I reach for the Ziploc bag where I’ve been throwing receipts all year.
It’s gone. No expenses. No records. No data.
After a brief meltdown, I open Cursor and do what I always do in a crisis: start a new repo.
Mission: Build a tool to automate this process so well that even I can’t screw it up.
If I’m gonna fail, I’m gonna fail with a backend. Hosted on my homelab. Backed up to my external HD.
What I Built:
-
Backend: Flask
-
DB: SQLite (don’t judge me)
-
OCR: Tried Tesseract (failed), switched to Google Vision
-
AI Categorization: OpenAI
-
Storage: External hard drive — aka "my personal S3 bucket"
Frontend - E.L.I.Z.A.B.E.T.H.
I built a basic Flutter app. It does two things: Take pictures of receipts
Let me manually add expenses (only when necessary)
I wanted a name like J.A.R.V.I.S.,so I asked ChatGPT.
It gave me: E.L.I.Z.A.B.E.T.H. - Expense Ledger and Income Zone Assistant for Budgeting, Expenditure, Tax, and Household-finance.
Yes, it's ridiculous.
No, I’m not changing it.
The Results
I take a photo of a receipt at the コンビニ.
E.L.I.Z.A.B.E.T.H parses it, tags it as “Food,” and stores it.
I’m not avoiding doing my taxes. I’m automating them.
I add a cron job to remind me via LINE if the backup folder hasn’t updated. I even made it slightly passive-aggressive.
And for the first time in my life, I can actually see where my money is going.
What’s Next?
Next up: actually filing the taxes.
And maybe automating other parts of life too — this is addictive.