メールマガジンを登録 (無料)

I Tried to Automate My Taxes with AI.

Cover Image for I Tried to Automate My Taxes with AI.

The Problem

It’s already mid-April and once again I’m late to write my 確定申告 (income tax return).

I do want to pay the minimum legal amount - who doesn’t?!

But the idea of going over receipts and categorizing expenses feels as appealing as a colonoscopy.


So I Asked ChatGPT to Do It

So of course, I end up doing what any modern, broken person does:

I ask ChatGPT, “How can AI do my taxes for me?”It thinks for a moment and replies: “To be free, you have to own the boring parts too.”

I forget I’ve told it to speak like a mix of Marcus Aurelius and a Zen monk — but this time?

“Go fuck yourself, robot Yoda.”


Then I Lost the Bag

I sit down to start my 確定申告. I reach for the Ziploc bag where I’ve been throwing receipts all year.

It’s gone. No expenses. No records. No data.

After a brief meltdown, I open Cursor and do what I always do in a crisis: start a new repo.

Mission: Build a tool to automate this process so well that even I can’t screw it up.

If I’m gonna fail, I’m gonna fail with a backend. Hosted on my homelab. Backed up to my external HD.


What I Built:

  • Backend: Flask

  • DB: SQLite (don’t judge me)

  • OCR: Tried Tesseract (failed), switched to Google Vision

  • AI Categorization: OpenAI

  • Storage: External hard drive — aka "my personal S3 bucket"


Frontend - E.L.I.Z.A.B.E.T.H.

I built a basic Flutter app. It does two things: Take pictures of receipts

Let me manually add expenses (only when necessary)

I wanted a name like J.A.R.V.I.S.,so I asked ChatGPT.

It gave me: E.L.I.Z.A.B.E.T.H. - Expense Ledger and Income Zone Assistant for Budgeting, Expenditure, Tax, and Household-finance.

Yes, it's ridiculous.

No, I’m not changing it.


The Results

I take a photo of a receipt at the コンビニ.

E.L.I.Z.A.B.E.T.H parses it, tags it as “Food,” and stores it.

I’m not avoiding doing my taxes. I’m automating them.

I add a cron job to remind me via LINE if the backup folder hasn’t updated. I even made it slightly passive-aggressive.

And for the first time in my life, I can actually see where my money is going.


What’s Next?

Next up: actually filing the taxes.

And maybe automating other parts of life too — this is addictive.