FinBERT, BloombergGPT & co: a survey of finance-specialised language models

In brief. General-purpose language models (BERT, GPT) have a poor grasp of the vocabulary, conventions and nuances of the financial domain. Since 2019, a family of specialised models, from FinBERT to BloombergGPT by way of FinGPT and FinMA, has emerged to bridge this gap. This article offers a comparative, pedagogical review of these models: their architectures, training corpora, performance and limitations.

Why “finance” language models?

Natural language processing (NLP) was transformed by the Transformer architecture introduced by Vaswani et al. (2017)¹, then by pre-trained models such as BERT (Devlin et al., 2019)² and the GPT family (Brown et al., 2020)³. Trained on general-purpose corpora (Wikipedia, Common Crawl, books), these models excel at cross-domain tasks.

But financial language has characteristics that challenge these general-purpose models:

• A specialised, polysemous vocabulary. Terms such as liability, bull, bear, short, maturity, provision or exposure carry a technical meaning far removed from their everyday use.
• Inverted sentiment semantics. In finance, “debt is rising” is not neutral: context determines whether it is positive or negative. Malo et al. (2014) formalised this difficulty with the Financial PhraseBank dataset⁴.
• Long, structured documents: annual reports, prospectuses, earnings releases, earnings call transcripts.
• High reliability stakes: a misinterpretation can have direct regulatory or financial consequences.

The guiding idea is domain adaptation. Gururangan et al. (2020), in “Don't Stop Pretraining”⁵, showed empirically that continued pretraining on a domain corpus (domain-adaptive pretraining) significantly improves downstream performance. The financial models presented below are all illustrations of this principle.

1. FinBERT: the first wave (2019–2020)

The name “FinBERT” actually covers several distinct models published independently, which is a frequent source of confusion.

1.1 FinBERT (Araci, 2019)

The first version, proposed by Dogu Araci⁶, starts from BERT and applies continued pretraining on the TRC2 financial corpus (Reuters), followed by fine-tuning on the Financial PhraseBank for sentiment analysis. The result: clear gains over general-purpose BERT on financial sentiment classification. It is the most cited and most widely used version in practice.

1.2 FinBERT (Yang et al., 2020)

Yang, Uy and Huang⁷ propose a variant trained on a larger financial corpus (10-K/10-Q filings, earnings call transcripts, analyst reports), evaluated on several tasks. This version emphasises building a large-scale financial corpus.

1.3 FinBERT (Liu et al., 2020)

Liu et al.⁸ present, at IJCAI 2020, a “FinBERT” pretrained end-to-end on financial texts with self-supervision tasks adapted to the domain, for financial text mining.

Key takeaway. “FinBERT” is not one model but a family. Always specify the author and year. An often-overlooked point: these models are BERT-type encoders, designed for classification and extraction; not, natively, for text generation or for semantic search (retrieval), which require additional adaptation.

2. Benchmarks: FLUE, FLANG and domain evaluation

Comparing models requires standardised evaluation sets. Shah et al. (2022) introduced FLUE (Financial Language Understanding Evaluation), a set of reference tasks, along with the FLANG model⁹. FLUE gathers varied tasks: sentiment analysis, headline classification, financial named-entity recognition, and structured question answering.

In finance, these benchmarks play the role that GLUE¹⁰ plays for general-purpose NLP: they enable reproducible, objective comparisons between models.

3. BloombergGPT: scaling up to generation (2023)

With BloombergGPT (Wu et al., 2023)¹¹, the paradigm shifts in scale. It is a generative (decoder-only) model with 50 billion parameters, trained on a mixed corpus of about 700 billion tokens: ~363 billion from Bloomberg's proprietary financial data (“FinPile”) and ~345 billion from public general-purpose data.

Key points:

• A hybrid approach: mixing financial and general-purpose data preserves cross-domain capabilities while specialising the model. It is a large-scale illustration of the domain-adaptation principle.
• Performance: BloombergGPT outperforms general-purpose models of comparable size on financial tasks, while remaining competitive on general-purpose benchmarks.
• A major limitation: the model and its data are proprietary and not open, which limits its reproducibility and community adoption.

BloombergGPT marked a turning point: it demonstrated that a large language model dedicated to finance delivered measurable value, paving the way for open alternatives.

4. FinGPT and FinMA: the open-source response (2023)

In response to BloombergGPT's closed nature, several open-source initiatives emerged.

4.1 FinGPT

FinGPT (Yang, Liu and Wang, 2023)¹² adopts a different philosophy: rather than retraining a giant model from scratch (costly and hard to reproduce), it builds on open LLMs (such as LLaMA¹³) and adapts them via efficient fine-tuning techniques such as LoRA (Low-Rank Adaptation, Hu et al., 2021)¹⁴. The emphasis is on up-to-date data pipelines and a reduced adaptation cost.

4.2 PIXIU / FinMA

The PIXIU project (Xie et al., 2023)¹⁵ offers the FinMA model, together with a financial instruction dataset and an evaluation benchmark (FLARE). The goal is to provide an open and reproducible ecosystem: model, instruction data and evaluation protocol.

These works illustrate a deeper trend: lightweight adaptation (instruction-tuning, LoRA) rather than massive pretraining, to democratise access to financial models.

5. Do you need a specialised model? The GPT-5, Claude vs. dedicated models debate

The arrival of very powerful general-purpose LLMs (GPT-5, Claude, etc.) has revived a legitimate question: is a specialised model still necessary?

Several studies, including Li et al. (2023)¹⁶, evaluate general-purpose LLMs on financial text analysis tasks. Their conclusions converge on a nuanced answer:

• Large general-purpose models achieve good zero-shot performance on simple tasks (sentiment, classification).
• But specialised or finely adapted models retain an edge on technical tasks, fine-grained terminology and, above all, inference cost and deployment control (sensitive data, latency, sovereignty).

The choice therefore depends on context: volume, data sensitivity, cost and latency constraints, target language.

Comparison table

Sizes are indicative and depend on the published variants.

How to choose: practical criteria

For a practitioner, choosing a financial model rarely comes down to “the biggest” or “the newest”. A few decisive criteria:

1. Task type. Classification/extraction → a FinBERT-style encoder is often enough and far cheaper. Generation/synthesis/reasoning → a generative model is necessary.
2. Language. Most of these models are English-language. For French or other languages, you must turn to multilingual models or consider a dedicated adaptation, a major and still under-explored research issue.
3. Openness and sovereignty. An open model allows deployment on controlled infrastructure, essential for sensitive financial data.
4. Inference cost and latency. A 50-billion-parameter model has operating costs on a completely different scale from a 110-million-parameter encoder.
5. Data freshness. A frozen model quickly becomes outdated in a domain where information evolves daily, hence the value of continuous-adaptation approaches.

Conclusion

In less than five years, financial language models have moved from FinBERT, an encoder specialised in sentiment analysis, to large-scale generative systems such as BloombergGPT, and then to an opening movement driven by FinGPT and FinMA. Two lessons stand out:

• Domain adaptation works: specialising a model on financial data delivers a measurable gain, consistent with the domain-adaptive pretraining principle.
• There is no universally superior model: the right choice depends on the task, the language, the budget and the deployment constraints.

The next challenges concern multilingualism (beyond English), continuous adaptation to recent data, and reliability evaluation in high-stakes contexts. All of these will be the subject of future articles.

References

Footnotes

1. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS. arXiv:1706.03762 ↩

2. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT. arXiv:1810.04805 ↩

3. Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS. arXiv:2005.14165 ↩

4. Malo, P., Sinha, A., Korhonen, P., Wallenius, J., & Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4), 782–796. arXiv:1307.5336 ↩

5. Gururangan, S., et al. (2020). Don't Stop Pretraining: Adapt Language Models to Domains and Tasks. ACL. arXiv:2004.10964 ↩

6. Araci, D. (2019). FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv:1908.10063 ↩

7. Yang, Y., Uy, M. C. S., & Huang, A. (2020). FinBERT: A Pretrained Language Model for Financial Communications. arXiv:2006.08097 ↩

8. Liu, Z., Huang, D., Huang, K., Li, Z., & Zhao, J. (2020). FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining. IJCAI. ↩

9. Shah, R. S., et al. (2022). When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain. EMNLP. arXiv:2211.00083 ↩

10. Wang, A., et al. (2018). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. ICLR Workshop. arXiv:1804.07461 ↩

11. Wu, S., et al. (2023). BloombergGPT: A Large Language Model for Finance. arXiv:2303.17564 ↩

12. Yang, H., Liu, X.-Y., & Wang, C. D. (2023). FinGPT: Open-Source Financial Large Language Models. arXiv:2306.06031 ↩

13. Touvron, H., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 ↩

14. Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 ↩

15. Xie, Q., Han, W., Zhang, X., Lai, Y., Peng, M., Lopez-Lira, A., & Huang, J. (2023). PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance. NeurIPS Datasets and Benchmarks. arXiv:2306.05443 ↩

16. Li, X., Zhu, X., Ma, Z., Liu, X., & Shah, S. (2023). Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks. arXiv:2305.05862 ↩