TEXMET: A Deep Learning Benchmark for Archaeological Textile Reconstruction

Zafar, Muhammad Hamza; Gigilashvili, Davit; Hardeberg, Jon Yngve

doi:10.1145/3846195

Overview

Teaching puzzle solvers about textiles

Reconstructing fragmented historical textiles is slow, manual work for archaeologists. AI puzzle solvers look promising, but they are trained on natural photographs and struggle with the faded colors, worn fibers and missing parts of ancient fabric: a classic domain gap.

To close it we introduce TEXMET, the first benchmark dataset designed for textile reconstruction: 18,644 high-resolution textile and tapestry images from the Metropolitan Museum of Art's Open Access collection, curated with CLIP embeddings, UMAP clustering, similarity search and human review.

We study how state-of-the-art puzzle solvers perform on textiles, how much domain-specific fine-tuning on TEXMET helps, how missing content and inpainting affect them, and what Large Vision Models (LVMs) can contribute.

The problem

The Oseberg find

The Oseberg ship burial (834 CE, Norway) held some of the finest textiles of the Viking Age. Today they survive as hundreds of fragile, degraded fragments at the Museum of Cultural History in Oslo, and nobody knows exactly how they originally fit together.

Oseberg textile fragments. Photos: G. A. Pantos, Museum of Cultural History, UiO

A scrambled Oseberg fragment reassembled by an AI puzzle solver — The task: a fragment photo cut into a 3×3 puzzle, shuffled, and reassembled by a solver

Oseberg ship illustration — The Oseberg ship (context illustration, Ulleland)

The dataset

Building TEXMET

No existing dataset fit: textile collections show modern fashion, fragment datasets show frescoes, and puzzle benchmarks use everyday photos. So we built one from The Met's open collection.

1 · Harvest

55,007images via the Met API, ~3 days

2 · Select

27,360objects, one main image each

3 · Embed & map

CLIP + UMAPsimilar images cluster, outliers stand out

4 · Review

−8,716armor, loose threads, near-empty images, duplicates

Result

18,644train 14,915 · val 1,864 · test 1,865

Final curated dataset in FiftyOne — The final curated set: 18,644 samples

Examples of removed images such as helmets filed as textiles — Not everything was a textile: helmets filed among the lace

Distribution across time periods — Time periods

Object names distribution — Object types

Load it in one line

All 18,644 images with Met metadata and the exact train / validation / test splits from the paper, as Parquet on Hugging Face. CC0.

Open on Hugging Face

from datasets import load_dataset

ds = load_dataset("hzafar/TEXMET")
ex = ds["train"][0]
ex["image"], ex["title"]

The solvers

Models & architectures

Three state-of-the-art puzzle solvers, all trained on ImageNet, reproduced and evaluated on textiles.

CVPR 2024

JPDVT

A diffusion vision transformer: starts from a noisy guess of where pieces go and refines it step by step. Robust to missing patches.

2025

FCViT

A vision transformer that predicts each piece's position directly by coordinate regression.

CVPR 2024

DiffAssemble

A graph-based diffusion model that reasons about piece-to-piece relationships; it can handle rotation, missing pieces and variable puzzle sizes.

ASUKA inpainting

Fills in missing regions of damaged images before puzzle solving.

GPT-4o and Gemini

Large Vision Models explored for tile arrangement, narrative generation and semantic retrieval.

Results

Fine-tuning on TEXMET closes the gap

Puzzle accuracy: the share of 3×3 puzzles with every piece in the right place. Trained on ImageNet only, the solvers struggle on textiles and on real Oseberg fragments. Fine-tuned on TEXMET, DiffAssemble more than doubles its Oseberg score. Oseberg is never used for training.

Model	ImageNet	TEXMET	Oseberg (45 fragments)
JPDVT (ImageNet-trained)	67.9%	32.9%	20.0%
FCViT (ImageNet-trained)	68.8%	8.3%	24.4%
DiffAssemble (ImageNet-trained)	56.6%	65.1%	26.7%
DiffAssemble fine-tuned on TEXMET	–	85.9% +20.8	64.4% +37.8

Textile features, not just outlines

On square crops from inside each Oseberg fragment (no outline), the fine-tuned model still beats the baseline: 39.8% vs 27.7%.

Robust to holes

On ImageNet photos with a third hidden by a real fragment-shaped mask, DiffAssemble barely moves while FCViT drops 24.5 points. Inpainting recovers part of the loss.

Fine-tuning isn't automatic

JPDVT forgot what it knew (catastrophic forgetting) and FCViT barely adapted; DiffAssemble was carried forward.

Gallery

DiffAssemble success examples — DiffAssemble reproduction (CelebA): successes

DiffAssemble failure examples — DiffAssemble reproduction (CelebA): failures

DiffAssemble with 30% missing pieces — DiffAssemble reproduction (CelebA): 30% of pieces missing, multiple sizes

DiffAssemble solving a 12x12 puzzle with rotations — DiffAssemble solving a 12×12 puzzle with rotations, step by step

FCViT perfect versus failed example — FCViT: perfect vs failed

Inpainting example — Masking and inpainting (ASUKA)

Oseberg benchmark — DiffAssemble on the Oseberg fragments

Tools

PuzzleSolver interface

An interactive tool to scramble images, run the solvers and inspect their reconstructions, including on Oseberg fragments. Code on GitHub.

Talk

GCH 2026, Barcelona

Presented at the Eurographics Workshop on Graphics and Cultural Heritage, ACM JOCCH Special Issue on Visual Heritage session, 3 November 2026.

Click into the slides and use the arrow keys; press F for full screen.

Open the slides

Takeaways

Conclusions and lessons

Conclusion

We introduced TEXMET, the first large-scale benchmark dataset for textile reconstruction.
DiffAssemble fine-tuned on TEXMET gives the best results on real Viking textile fragments (26.7% → 64.4% on Oseberg).
Models must learn the textile patterns themselves, not shortcuts like outlines: our crop experiments test exactly that.
Large Vision Models are useful as assistive tools (metadata, semantic retrieval), not for solving the puzzle itself.

Future work

Move beyond grid puzzles to irregularly shaped fragments, for example with graph representations that encode boundary shapes or attention focused on contour matching. Build larger, more diverse archaeological datasets for pre-training. Explore hybrids where an LVM's semantic understanding guides the spatial search of a solver like DiffAssemble.

Reproduction challenges

Reproducing published models took far longer than expected: the 4–6 weeks per model initially planned became 8–12 weeks for problematic models like FCViT. That bottleneck shaped the research, which is why we share code and data to help others avoid the same pitfalls.

Compute

Most experiments ran on NTNU's IDUN HPC cluster on H100 and A100 nodes (4 GPUs, 32 CPUs, 200 GB RAM per node); DiffAssemble needed A100 nodes for older PyTorch compatibility. About 3,000 GPU hours in total, plus smaller tests on a local RTX 4090. Slurm scripts are in hpc-jobs-ntnu. We thank IDUN NTNU HPC for the resources.

Cite

Citation

If you use TEXMET, please cite the paper:

@article{zafar2026texmet,
  author  = {Zafar, Muhammad Hamza and Gigilashvili, Davit and Hardeberg, Jon Yngve},
  title   = {{TEXMET}: A Deep Learning Benchmark for Archaeological Textile Reconstruction},
  journal = {ACM Journal on Computing and Cultural Heritage},
  year    = {2026},
  doi     = {10.1145/3846195},
  eprint  = {2611.04734},
  archivePrefix = {arXiv}
}

Code & data

All resources

TEXMET on Hugging FaceFull dataset, Parquet, splits TEXMET on GitHubDownload scripts and metadata PaperACM JOCCH, 2026 arXiv:2611.04734Open-access preprint JPDVT-MT-NTNUJPDVT reproduction FCViT-MT-NTNUFCViT reproduction DiffAssembleFine-tuning on TEXMET asuka-misatoInpainting fork PuzzleSolverUIInteractive tool hpc-jobs-ntnuSlurm scripts GitHub topicthesis-2025 Thesis project pageWork in progress GCH 2026 talkInteractive slides