How do Tabular Foundation Models work?

Jan Henrik Bertrand· · 5 min read· Practitioner's Guides

Tabular foundation models (TFMs) have taken the field of predictive AI by storm. For a decade tabular ML has looked something like this: fit a gradient-boosted tree ensemble (XGBoost, LightGBM, CatBoost) per dataset, then spend days tuning it. Then TFMs got introduced. A tabular foundation model (TFM) is a transformer-based model that was pretrained on millions of synthetic datasets to predict on any new tabular dataset without re-training or fine-tuning. It does so using in-context learning (ICL).

The revolution started with TabPFN (Hollmann et al., Nature 2025), the first TFM which beat tuned baselines on small tables in seconds, and practical at scale when TabICLv2 (2026) extended the same idea to datasets with up to 100k rows. In this article you will learn how TFMs work, how they are pre-trained and what you should know before using them.

The inner workings of a TFM

In essence, a TFM stands on two ideas: an attention mechanism that allows the TFM to contextualize each cell in a table within its row and column, and in-context learning, which turns prediction into a single forward pass. Both steps are explained in the following section, based on the TabICLv2 architecture.

Two-way attention: contextualizing each cell within the table

Text has one axis along which tokens have to be related to each other taking the order into account. A table on the other hand has two axes and the ordering of columns and rows bears no information. That changes how the attention mechanism of the Transformer has to be applied. TFMs use socalled Set-Transformers: all values in a column/row are the set that is fed into Set-Transformer layers. Some of those layers are applied along the column-axis and other along the row-axis. During pre-training the TFM learns to use the attention mechanism in order to put the value of a given cell into the context of its row and column. Thereby it captures interactions between features and samples. The result is invariant to shuffling rows or columns, i.e. exactly the symmetry of a table demands.
Fig. 1 below shows how one layer of a TFM consisting of a column-wise and row-wise Set-Transformer relates each cell in a table to each other cell.

x₁ x₂ x₃ x₄ x₅ x₆ y test train ? ? cell in focus — one turn each 1 · column step its feature thaws across samples 2 · row step its sample thaws across features translucent copies ≙ q·k·v — each round trip nudges the focus cell after each test row has attended, its masked target resolves
Fig. 1 — Two-way attention: each cell takes a turn in focus (red): its value is related to all other values in its column, then the same happens along its row; every round trip refines the cell's representation, which persists. As each test row finishes attending, its masked target resolves into the prediction.
Note: this is a schematic depitction that omits a few details and architectural nuances.

In-context learning: your dataset is the prompt

At prediction time, a TFM does what a large language model does with a prompt, but its tokens are rows. Your labeled rows (train set) form the context; your query rows (test/inference set) ride along with their labels masked. During the forward pass, each query row gets a full predictive distribution: class probabilities for classification, quantiles for regression. Nothing is fitted, so fit() in these libraries mostly just encodes and caches the context.

Robustness to missing values, outliers, and uninformative columns comes for free, because the prior contained all three. The costs are equally structural: the whole context sits in memory at inference, and per-prediction latency and memory consumption is far above that of a compiled tree ensemble.

x₁ x₂ x₃ y test train ? TFM layer ×12 1 · cross-feature attention 2 · cross-sample attention x₁ x₂ x₃ y test train MLP p(ŷ) binned over y predictive distribution “?” are the labels to be predicted
Fig. 2 — Inference as in-context learning. The query row enters with its label masked. In every layer its cells first attend within the sample (feature attention), then across every sample of the context (sample attention) — twelve such layers refine the masked label's hidden representation. An MLP head finally turns that representation into a piecewise-constant (binned) predictive distribution over y.

How to pre-train a TFM

To pre-train a TFM, we need to instill the mechanism to detect relationships between cells in the dataset and the labels into the TFM. That is done by sampling synthetic tabular datasets and pre-training the TFM on those. In order to sample datasets with rows and columns related to each other, we need to sample the dataset from some kind of causal structure: a Structural Causal Model (SCM). An SCM is a tree-like graph structure that models dependencies between features through its graph structure. Several layers can make the relations between features more abstract. For a visualization on how such a dataset is generated, see Fig. 3 below.
Once the dataset is generated, for part of the samples, we mask the label — those become the query samples and the rest are the context samples. Finally we backpropagate using the loss of the query set samples.

Do this on the order of 10⁸ times and something useful happens: the network cannot memorize datasets, so it is forced to learn a procedure for mapping labeled examples to predictions. Formally, this objective makes the model approximate Bayesian posterior-predictive inference under the prior — averaging over all data-generating explanations that are consistent with the rows it sees. A TFM is best understood not as a trained model but as a learned learning algorithm, compressed into fixed weights.

target features regression binary multiclass TFM pretrain y test train ? ? edge shade ≙ strength of dependence sampled ~10⁸ times for pretraining · none of the tables real · none of them yours
Fig. 3 — The synthetic prior. Every row is one pass through a random structural causal model: a target activation is set, propagates through the graph (edge shade = dependence strength), and the feature-layer activations drop below as the row's values — train rows filling from the bottom, held-out test rows above. Each finished dataset docks under the TFM for one training pass, then the next world is sampled; the target head alternates between regression, binary, and multiclass tasks.

What you should know before using a TFM

Ok now we know how a TFM works, where it outperfoms and how its pre-trained. However, before you can start integrating one into your pipeline you should know about a few central questions: (1) Licensing – are you allowed to use it in a commercial setting? (2) Scaling – what infrastructure do you need to host one? and (3) comparison – is a TFM is the right choice for your task?

(1) Licensing. New tabular foundation models are released on a monthly basis. TabPFN in Nature (January 2025), TabPFN-2.5 (November 2025), TabICLv2 (February 2026), TabPFN-3 (May 2026), TabFM (June 2026). Licensing has tightened with it over time. TabPFN-3's weights are free to download but licensed for research, evaluation, and internal testing only; TabFM's weights are non-commercial even though its code is Apache-2.0. The trap: the best model and the best legally deployable model change on different schedules, and quietly upgrading a revenue-generating pipeline to a restricted checkpoint can be a license violation and can be followed by serious legal consequences. Commercial use means license review on every release.

(2) Scaling. TFMs are transformers: the entire context sits in memory at inference, which often requires a GPU to handle the computational and memory load. The first version of TabPFN attends over every cell of the table and tops out around 10,000 rows; row-compression architectures (TabICLv2, TabFM) first squeeze each sample into one vector so attention runs over n rows instead of n×m cells, stretching the practical ceiling to roughly 100k rows on a single large GPU. Even then, per-prediction latency stays well above a compiled tree ensemble's. When deploying a TFM in production, you should think about GPU provisioning, context caching, batching, and re-benchmark every new checkpoint for your task.

(3) Comparison. TFMs lead the public benchmarks on small and mid-sized tables — but leading a benchmark is an average, not a guarantee. Their win rate against tuned gradient boosting is nowhere near 100%, and on large datasets, heavily categorical data, or latency-critical serving, classical models still come out ahead often. The only way to know what wins on your dataset is to run the comparison yourself: several model families, tuned fairly, evaluated on your metric.

My personal recommendation

Spending weeks on training and tuning all kinds of models to find the best one for a given task, setting up and paying for GPU infrastructure to deploy a TFM at scale and the headache regarding the licensing are not worth it. Instead I would highly recommend using an API like QBrain API which is free for many datasets and comparatively cheap for bigger ones. On top of that, the QBrain API serves Qombra's tabular foundation model that is actively being improved so you always receive frontier predictions without benchmarking and exchanging production models against new ones. Its performance and ease of use make it a no-brainer for production workloads.

Frequently asked questions

Do tabular foundation models replace gradient boosting?

The short answer is no. While TFMs outperform classical methods on small and medium-size datasets on benchmarks, they don't get even close to a 100% winrate. So you need to try all sorts of different models to get the best performance or use QBrain to get the best possible performance for your dataset through our proprietary predictive engine.

Can I use TabPFN v3 or TabFM weights in a commercial product?

Not under their default licenses. TabPFN-3 weights are free to download but restricted to research, evaluation, and internal testing; TabFM's weights carry a non-commercial license even though its code is Apache-2.0. The weights are what make predictions, so a permissive code license does not help. Managed APIs such as the QBrain API avoid the issue: you consume predictions as a service instead of deploying restricted weights.

Do tabular foundation models need a GPU?

For small tables, CPU inference works but is noticeably slower. Anything beyond a few thousand rows effectively wants a GPU, and memory grows with the context you feed in — one reason many teams prefer calling a hosted API over operating GPU serving themselves.

How much data can a tabular foundation model handle?

The Nature-published TabPFN targets up to about 10,000 samples and 500 features. Row-compression architectures moved the ceiling: TabICLv2 performs well on datasets with around 100k rows and up to 500 features on a single 50 GB GPU. Above that scale, classical methods or sampling strategies still apply.