AI moon crater catalogs fall short under uniform testing
A Southwest Research Institute-led study found reported accuracy for several AI lunar crater catalogs dropped sharply under common scientific criteria.
By Tom Brennan · Health & Medicine Correspondent
3 min read
A Southwest Research Institute-led team found that several AI-built lunar crater catalogs performed far worse when judged by the same criteria, according to a study published in The Planetary Science Journal. The finding matters for planetary science because researchers use crater counts and measurements to estimate the ages and histories of the moon and other solid worlds.
The study compared eight automated lunar crater databases that cover the whole moon or large areas of it, SwRI said. Stuart J. Robbins of SwRI’s Solar System Science and Exploration Division led the work with SwRI researcher Rachael H. Hoover.
According to SwRI, crater catalogs record the positions, sizes and other features of impact structures. Scientists use those records to study how planetary surfaces changed over time.
Researchers estimate surface ages by counting craters because small asteroids hit planetary bodies at a roughly steady rate, SwRI said. Under that method, surfaces with more craters are generally treated as older than surfaces with fewer craters.
AI results changed under stricter matching rules
Robbins and Hoover tested the eight AI-generated catalogs against a large manual lunar crater catalog that Robbins spent years compiling, according to SwRI. The researchers applied the same matching rules across the AI catalogs rather than relying on each catalog’s published performance measures.
The study found that an automated crater detection result can appear strong under some computer vision metrics while still being scientifically weak, SwRI said. The team said useful crater records must place craters correctly and measure their diameters accurately.
SwRI said many of the catalogs’ published performance figures fell when the team used criteria tied to how consistently human crater analysts identify the same features. In some cases, the values declined by more than a factor of 10, according to the institute.
Robbins said AI can help with repetitive scientific work, including data collection, but researchers should not assume an automated crater catalog is ready for research only because its reported metrics look strong. He said errors such as shifted, duplicated or mis-sized craters can affect studies that depend on those measurements.
As one example, Robbins said a duplicated crater count could distort a model age for a surface. If the model age depends on a certain number of craters and the catalog counts the same craters more than once, the resulting age estimate could be inflated, according to SwRI’s summary of the study.
Single scores can hide weaknesses
The researchers also found that one overall accuracy number can conceal uneven performance, SwRI said. A catalog may work better for some crater sizes than for others, which means it could be useful for one scientific question and unreliable for another.
Robbins said crater diameter is a key factor in judging an automated catalog. According to SwRI, breaking results down by crater size showed weaknesses that broad summary numbers did not reveal.
Hoover said the findings point to a need for standard benchmarks, clearer reporting of matching rules and independent validation before AI crater catalogs are used widely in scientific analysis. SwRI said the study does not argue against using AI in planetary science.
Robbins said AI may eventually change crater cataloging and save researchers years of work. For now, he said scientists need to understand where the tools succeed, where they fail and whether their performance is good enough for the research question at hand.
The study, “A Comparison of Lunar AI-Based Crater Databases Using Uniform Criteria,” was published in The Planetary Science Journal, according to SwRI.
This story draws on original reporting from Phys.org.