lexglue-legalbert-unfair-tos

Legal-BERT fine-tuned on LexGLUE UNFAIR-ToS — matches published benchmark

Model Description

This model is fine-tuned on the LexGLUE UNFAIR-ToS benchmark dataset to detect unfair clauses in Terms of Service documents.

Base Model: nlpaueb/legal-bert-base-uncased
Training: Standard BCEWithLogitsLoss, lr=3e-5, batch_size=8, linear scheduler, up to 20 epochs with early stopping (matching Chalkidis et al., 2022)

Performance

Evaluated on the official LexGLUE test set (1,607 samples) using the paper's evaluation methodology (includes implicit "fair" class in micro-F1 computation).

Metric	Score
Micro-F1 (LexGLUE method)	96.0
Macro-F1 (LexGLUE method)	84.1
Exact Match Accuracy	95.7%

Comparison with LexGLUE Leaderboard

Model	μ-F1	m-F1
Legal-BERT (LexGLUE paper)	96.0	83.0
CaseLaw-BERT (LexGLUE paper)	96.0	82.3
RoBERTa-large (LexGLUE paper)	95.8	81.6
lexglue-legalbert-unfair-tos (ours)	96.0	84.1

Risk Categories

The model classifies text into 8 types of potentially unfair clauses:

ID	Category	Description
0	Limitation of liability	Limits the provider's legal responsibility
1	Unilateral termination	Provider may terminate without clear cause
2	Unilateral change	Terms can change with minimal notice
3	Content removal	Provider may remove user content
4	Contract by using	Agreement implied by using the service
5	Choice of law	Specifies governing jurisdiction's law
6	Jurisdiction	Specifies where disputes are handled
7	Arbitration	Requires arbitration instead of court

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "Agreemind/lexglue-legalbert-unfair-tos"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

labels = [
    "Limitation of liability", "Unilateral termination",
    "Unilateral change", "Content removal",
    "Contract by using", "Choice of law",
    "Jurisdiction", "Arbitration",
]

text = "We may terminate your account at any time without notice."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

with torch.no_grad():
    probs = torch.sigmoid(model(**inputs).logits).squeeze()

for label, prob in sorted(zip(labels, probs), key=lambda x: x[1], reverse=True):
    if prob > 0.5:
        print(f"  {label}: {prob:.3f}")

Citation

If you use this model, please cite:

@article{chalkidis2022lexglue,
  title={LexGLUE: A Benchmark Dataset for Legal Language Understanding in English},
  author={Chalkidis, Ilias and Jana, Abhik and Hartung, Dirk and Bommarito, Michael and Androutsopoulos, Ion and Katz, Daniel Martin and Aletras, Nikolaos},
  journal={arXiv preprint arXiv:2110.00976},
  year={2022}
}

License

MIT