Proliferation of national research evaluation procedures is arguably one of the main factors shaping the contemporary institutional landscape of science in Europe. Consequently, we observe a growing interest in research evaluation’s impact on the behaviour of both researchers and research institutions. So far, however, we have limited understanding of a social process behind the construction of research evaluation procedures themselves. Hidden behind closed doors of policy-forming chambers and committees, they often remain beyond researchers’ view. In effect, research evaluation forms a black box, which can be studied primarily through its inputs and outputs. In an attempt to open this black box, this study follows in the direction set by the sociology of quantification and asks: what shapes the production of numbers in the context of research evaluation? It posits this question in the context of the Polish science system, as it takes advantage of its unique character, i.e. high participation of the academic community in the creation of evaluation procedures. In particular, this study draws from the recent experience of creating the Polish Journal Ranking, a crucial policy instrument of  research evaluation system in Poland, which is studied in reference to two disciplines, biology and history. A mixed method approach is adopted, combining semi-structured interviews with actors of this process with a quantitative analysis of consecutive changes in the ranking on the different stages of its construction. This, in turn, sheds light on a complex social process of creating evaluative homogeneity out of disciplinary heterogeneity. The study further argues that this process is dictated by how actors position themselves in relation to two opposing forces: those favouring the homogenization of research and those maintaining its heterogeneity. As these forces remain imbalanced, the study concludes with a call for further exploration of the interplay between the forces of homogenization and heterogenization and how the tensions between them are mediated within a performance-based research funding system.