Improving Legal Document Labeling by Comparing Similar Sentences

Table of Links

Abstract and 1. Introduction

2. Related Work

3. Task, Datasets, Baseline

4. RQ 1: Leveraging the Neighbourhood at Inference

   4.1. Methods

   4.2. Experiments

5. RQ 2: Leveraging the Neighbourhood at Training

   5.1. Methods

   5.2. Experiments

6. RQ 3: Cross-Domain Generalizability

7. Conclusion

8. Limitations

9. Ethics Statement

10. Bibliographical References

4. RQ 1: Leveraging the Neighbourhood at Inference

In this section, we leverage the knowledge from semantically similar training instances directly during inference without extra training overhead. We interpolate the label distribution predicted by the baseline model with the distribution derived from the training instances similar to the test instance. This overcomes the problem of memorizing/learning rare patterns implicitly in the model parameters, thus enhancing the model’s ability to handle longtail cases (classes with few instances or rare patterns in frequent classes) especially in limited data settings. We explore three different methods to obtain the distribution from similar training instances.

4.1. Methods

4.1.1. Interpolation with kNN

In this method, we construct a datastore of training instances and then retrieve the nearest neighbours to the test instance for computing the interpolated label distribution during the inference.

Datastore Construction After training, we obtain contextualized representation ci of every sentence in each document of the training set using the trained model. We construct the datastore by a single forward pass over each training document. The datastore {K, V } is the set of all contextualized representation-rhetorical label pairs constructed from all the training examples D as:

Interpolation During inference time, we query the datastore using the contextualized representation of every sentence in the test document, to find the k-nearest neighbours N according to the euclidean distance. Then, we derive the distribution of labels pkNN using labels of the retrieved neighbours based on softmax of their negative distances, while aggregating probability mass for each label across all its occurrences in the retrieved neighbours (labels that do not appear in the retrieved N have zero probability). Intuitively, the closer a neighbor is to the test instance, the larger its weight is.

4.1.2. Interpolation with Single Prototype

Instead of storing all the training instances in the datastore, we seek to store one prototype for each label, which captures the essential semantics of various sentences for each rhetorical role, significantly reducing the datastore’s memory footprint. To create a prototype for each label, we calculate the average of the contextualized embeddings of sentences that share the same rhetorical role. Intuitively these prototypes can be assumed to be the center of clusters for different labels, surrounded by sentences expressing the same label in the embedding space. The interpolation process closely resembles the kNN approach (Eq. 5), with the key difference being that interpolation directly involves the prototypes, rather than a prior retrieval step.

4.1.3. Interpolation with Multiple Prototypes

Instead of using a single prototype for each rhetorical role, we suggest the use of multiple prototypes for each label. This choice is driven by the fact that instances with the same rhetorical role can exhibit distinct variations in expression, resulting in diverse contextual embeddings scattered across the embedding space. Averaging these embeddings into a single prototype might diminish specificity. Utilizing multiple prototypes allows us to effectively capture the intricate viewpoints within each label. To accomplish this, we cluster the instances belonging to each rhetorical role using k-means, yielding multiple prototypes for each label from the k centroids. The interpolation step remains similar (Eq.5), involving all these multiple prototypes without any retrieval step.