How do Tabular Foundation Models work?

Tabular foundation models (TFMs) have taken the field of predictive AI by storm. For a decade, tabular ML has looked something like this: clean your dataset, fit a gradient-boosted tree ensemble (XGBoost, LightGBM, CatBoost) for your task, then spend days optimizing hyperparameters. Recently, TFMs got introduced. A tabular foundation model (TFM) is a transformer-based model that was pretrained on millions of synthetic datasets to predict on any new tabular dataset without re-training or fine-tuning. It does so using in-context learning (ICL).

The revolution started with TabPFN (Hollmann et al., Nature 2025), the first TFM which beat tuned baselines on small tables, and became practical at scale when TabICLv2 (2026) extended the same idea to datasets with up to 100k rows. In this article you will learn how TFMs work, how they are pre-trained and what you should know before using them.

The inner workings of a TFM

In essence, a TFM stands on two ideas: an attention mechanism that allows the TFM to contextualize (i.e. relate) each cell in a table within its row and column, and in-context learning, which turns prediction into a single forward pass. Both steps are explained in the following section, based on the TabICLv2 architecture.

Two-way attention: contextualizing each cell within the table

Text has one axis along which tokens have to be related to each other taking the order into account. A table on the other hand has two axes and the ordering of columns and rows bears no information. That changes how the attention mechanism of the Transformer has to be applied. TFMs use so-called Set-Transformers: all values in a column/row are the set that is fed into Set-Transformer layers. Some of those layers are applied along the column-axis and other along the row-axis. During pre-training the TFM learns to use the attention mechanism in order to put the value of a given cell into the context of its row and column. Thereby it captures interactions between features and samples. The result is invariant to shuffling rows or columns, i.e. exactly the symmetry of a table demands.
Figure 1 below shows how one layer of a TFM consisting of a column-wise and row-wise Set-Transformer relates each cell in a table to each other cell.

In-context learning: your dataset is the prompt

At prediction time, a TFM does what a large language model does with a prompt, but its tokens are rows. Your labeled rows (train set) form the context; your query rows (test/inference set) ride along with their labels masked. During the forward pass, each query row gets a full predictive distribution: class probabilities for classification, quantiles for regression. Nothing is fitted, so fit() in these libraries mostly just encodes and caches the context.

Robustness to missing values, outliers, and uninformative columns comes for free, because the prior contained all three. The costs are equally structural: the whole context (i.e. your training data) sits in memory and gets run through the attention mechanism, which scales quadratically. Thus, the per-prediction latency and memory consumption is far above that of a compiled tree ensemble.

How to pre-train a TFM

To pre-train a TFM, we need to instill the mechanism to detect relationships between cells in the dataset and the labels into the TFM. That is done by sampling synthetic tabular datasets and pre-training the TFM on those. In order to sample datasets with rows and columns related to each other, we need to sample the dataset from some kind of causal structure: a Structural Causal Model (SCM). An SCM is a tree-like graph structure that models dependencies between features through its graph structure. Several layers can make the relations between features more abstract. For a visualization on how such a dataset is generated, see Figure 3 below.
Once the dataset is generated, for part of the samples, we mask the label. Those become the query samples and the rest are the context samples. Finally, we backpropagate using the loss on the query set samples.

Once we do this on the order of 10⁸ times and something useful happens: the network cannot memorize datasets, so it is forced to learn a procedure for mapping labeled examples to predictions. Formally, this objective makes the model approximate Bayesian posterior-predictive inference under the prior from pre-training. A TFM is best understood not as a trained model but as a learned learning algorithm, compressed into fixed weights.

What you should know before using a TFM

Now we know how a TFM works, where it outperfoms and how its pre-trained. However, before you can start integrating one into your pipeline you should know about a few central questions: (1) Licensing – are you allowed to use it in a commercial setting? (2) Scaling – what infrastructure do you need to host one? and (3) comparison – is a TFM the right choice for your task and if so, which one?

(1) Licensing. New tabular foundation models are released on a monthly basis. TabPFN in Nature (January 2025), TabPFN-2.5 (November 2025), TabICLv2 (February 2026), TabPFN-3 (May 2026), TabFM (June 2026), TabPFN-3.5 (September 2026). However, most of the aforementioned models are not permitted for commerical use as per their license in the code repository. For instance, TabPFN-3.5's weights are free to download but licensed for research, evaluation, and internal testing only; TabFM's weights are non-commercial even though its code is Apache-2.0. Hence, quietly upgrading a revenue-generating pipeline to a restricted checkpoint can be a license violation and can be followed by serious legal consequences. Commercial use requires reading licenses closely.

(2) Scaling. TFMs are transformers: the entire context sits in memory at inference, which often requires a GPU to handle the computational and memory load. That also often limits TFMs to outperforming only on benchmarks with mostly small datasets although row-compression architectures (TabICLv2) have made advances toward outperforming on larger datasets as well, stretching the practical ceiling to roughly 100k rows on a single large GPU. Even then, per-prediction latency stays well above a compiled tree ensemble's. When deploying a TFM in production on industry-scale datasets of several thousand or more samples, you should consider GPU provisioning, context caching, batching, and re-benchmark every new TFM as well as classical models for your task.

(3) Comparison. TFMs lead the public benchmarks on small and mid-sized tables — but leading a benchmark is an average, not a guarantee. The win rate of most open-source TFMs against other open-source methods is nowhere near 100%, and on large datasets, heavily categorical data, or latency-critical serving, open source TFMs like LimiX-2 can fall far behind what benchmarks suggest. The only way to know what wins on your dataset is to run the comparison yourself: several model families, tuned fairly, evaluated on your metric.

My personal recommendation

Spending weeks on training and tuning all kinds of models to find the best one for a given task, setting up, managing and paying for GPU infrastructure to deploy a TFM at scale and the licensing headache are not worth it. Instead, I would recommend using QBrain through Qombra's API, which is far cheaper and faster than any self-hosted TFM and performs better than any other predictive method (cf. Fig. 5). Qombra's tabular foundation model that is actively being improved so you always receive frontier predictions without benchmarking and exchanging production models against new ones.