The reliability of jurimetrics research

challenges in data collection and empirical case law analyses

Abstract

Objective: To underscore the necessity of a rigorous and specific methodology for validating the reliability of quantitative research within the legal field, while demonstrating how deficiencies in the data collection phase may compromise both the validity and the reliability of such research.
Methodology: Exploratory analysis with qualitative and quantitative approaches. The research was conducted by examining theoretical works that stipulate how jurimetric research should be carried out, and normative publications from the National Council of Justice and the Supreme Federal Court that regulate the subject. Following this, external case law systems of eight state courts were examined.
Results: The article demonstrates how these problems significantly hinder or even make impossible the quantitative empirical research of case law and statistically exposes the quality of indexing used in the search engines of case law from the State Courts of São Paulo, Rio Grande do Sul, Paraná, Pernambuco, Minas Gerais, the Federal District, Rio de Janeiro, and Amazonas.
Conclusions: The quantitative analysis revealed the problems arising from the poor indexing quality and lack of standardization across eight external Brazilian case law search databases, and methods were proposed to overcome these difficulties.