AI reads COBOL better than it writes Java

I have not migrated a COBOL estate. This piece is research, not experience, and I want that clear before the first claim rather than buried at the end.

What prompted it was the gap between two sets of numbers that are currently being published about the same activity.

On one side, vendor material describing million-line migrations in months rather than years, work that used to take fifty consultants three years now delivered in three to six. One vendor claims 99.5 percent logic preservation.

On the other, IBM's own research on enterprise-scale COBOL-to-Java translation, presented at ICSE, reporting a median structural quality score above 80 percent and a functional score above 75 percent.

Those are not describing different tools. They are describing the same class of work, and only one of them is being measured.

What a functional score of 75 percent means

Take the IBM figure at face value, because it is the trustworthy one. It comes from the organisation with the most to gain from a higher number, which is exactly why it is worth believing.

A median functional score above 75 percent means that on a typical module, something approaching a quarter of the behaviour is not faithfully reproduced. Not absent, which would be obvious. Different.

For most software, that is a bad but survivable starting point. For the systems that are still written in COBOL, it is not, because the reason they are still written in COBOL is that they calculate things people receive money from. Payments, claims, benefits, interest, premiums. The failure mode is not a crash. It is a slightly different number, produced confidently, at scale, discovered in an audit.

The 99.5 percent claim is doing something subtly different, and it is worth naming. It describes logic preservation validated through parallel testing: running both systems against the same transactions and comparing. That is a genuinely good technique, and the figure is a statement about the transactions that were tested. The business rules that only fire on a leap year, or above a threshold nobody has hit since 2019, are not in that sample. Those rules are most of what makes a COBOL estate frightening.

Where the evidence actually supports AI

This is where the research gets more interesting than the scepticism.

Every serious source on this, including the vendor material, says the same thing about the first stage: the hard part is understanding the source. HCLTech's pilot on a nine-million-line application put the effort into discovery and analysis, generating functional specifications, call flows and business rules before any translation. IBM's pipeline augments the model with program analysis rather than asking it to translate cold.

That is the part I would trust, and it is consistent with what I see in domains I do know. Reading a large unfamiliar codebase and producing an accurate account of what it does is something models are genuinely good at, because it is a comprehension task with the ground truth sitting right there in the source.

It is also, on a COBOL estate, the most valuable thing anyone could do. The business logic in a forty-year-old system is usually undocumented, its authors gone, and the organisation's actual understanding of its own rules is a folk memory held by three people near retirement. Recovering that into a written specification has standalone value even if not a line of Java ever gets generated.

Which is the argument I would make to a board: fund the comprehension, treat the translation as a separate decision taken afterwards with better information.

The part nobody is selling

The uncomfortable implication of a 75 percent functional score is that verification, not translation, is the budget line that matters.

If you accept that a model gets you to roughly three-quarters correct, then the project is not a translation project with a testing phase on the end. It is a verification project with a translation head start. The question stops being how fast the code can be produced and becomes how you establish that the produced code is right, given that the specification of correct is the old system's behaviour and nobody has written it down.

Parallel running is the honest answer, and it is expensive. Both systems live, every transaction through both, differences investigated rather than explained away. For a long time. That is the cost the three-to-six-month timelines are quietly leaving out, and it has not been reduced by AI at all.

There is a second-order version of the problem I find more interesting. When translation was slow and manual, its cost acted as a brake: you migrated the modules that justified the effort. If translation becomes nearly free, that brake is gone, and organisations will translate estates that would never have survived a business case. The verification bill arrives later and lands on someone else.

What I would want to see before believing more

A functional score is a proxy. What would change my reading is published data on defects found in production after an AI-assisted migration, at twelve and twenty-four months, from someone who is not selling the tooling. Migration defects surface slowly, at period end, at year end, in the audit.

Until that exists, the reasonable position is that AI has substantially improved one stage of COBOL modernisation, the stage where you work out what the system currently does, and has not yet moved the stage that decides whether the project succeeds.

If you are weighing up a modernisation business case and the vendor timelines look too good, I am happy to talk it through.