What is BERT and how are its embeddings used in finance?

Question

AcadiFi · Accepted Answer

BERT (Bidirectional Encoder Representations from Transformers, Devlin et al. 2018) is a Transformer encoder pre-trained on large text corpora using two self-supervised tasks:

1. **Masked Language Modeling (MLM)**: randomly mask 15% of tokens and train model to predict them. Forces bidirectional context understanding.
2. **Next Sentence Prediction (NSP)**: given two sentences, predict if B follows A. Teaches sentence-level relationships. (Later models like RoBERTa dropped NSP.)

Output: a contextual embedding vector (typically 768 dimensions for BERT-base) for each token. Unlike Word2Vec's static embeddings, BERT's embeddings depend on surrounding context - 'bank' in 'river bank' differs from 'investment bank.'

Usage patterns:
- **Feature extraction**: freeze BERT, use CLS token embedding as input to a classifier.
- **Fine-tuning**: add a task-specific head (classification, regression), train end-to-end on labeled data.

FinBERT (Araci, 2019): BERT pre-trained further on financial corpora (Reuters, earnings calls, analyst reports). Specialized for sentiment classification of financial text.

Finance applications:
- **Earnings call sentiment**: score each sentence as positive/neutral/negative, aggregate to signal.
- **Analyst report classification**: buy/hold/sell recommendation extraction.
- **Event detection**: M&A announcements, guidance changes, regulatory filings.
- **ESG scoring**: classify sustainability disclosures.
- **RAG systems**: generate embeddings for 10-K chunks, retrieve relevant passages for question answering.

Typical workflow at a buy-side shop:
1. Scrape earnings transcripts via Capital IQ.
2. Split into sentences, embed with FinBERT.
3. Aggregate sentiment per call; cross-sectional rank; build long/short portfolio on top decile minus bottom decile.

Limitations: fixed input length (512 tokens for BERT-base), expensive fine-tuning, domain shift issues (general BERT underperforms on finance jargon).

What is BERT and how are its embeddings used in finance?

Master Level II with our CFA Course

Related Questions

Practice Questions