Learning to Directly Maximize Evaluation Metrics

Abstract

In particle physics, unraveling the mysteries of the Higgs Boson is a crucial task. In particular, gaining a precise understanding of the tt̄bb̄ process, which is related to the tt̄H(bb̄) and the associated issues, is closely related to elucidating the properties of the Higgs Boson. Therefore, in the tt̄bb̄ event, classifying (or identifying) b-jets originating from top quark decay and the additional b-jets generated from gluon splitting is critical. Since no simple physical rules exist for this identification problem, previous studies have employed machine-learning approaches based on simulated data. Existing methods train without individually structuring each tt̄bb̄ event; instead, they perform binary classification on all b-jets and the additional b-jets, minimizing the binary cross-entropy loss. However, the performance metric Matching Efficiency of the prediction model evaluates the ability to identify the number of additional b-jets for every structured tt̄bb̄ data. Given the metric’s evaluation purpose, we propose reconstructing the data into the tt̄bb̄ structure. Furthermore, we introduce the surrogate loss function to directly maximize this evaluation metric for achieving improved performance compared to existing approaches. To compare the concepts of the previous and proposed methods, we performed synthetic data experiments to evaluate two loss functions. We generated synthetic data that preserved the structural characteristics, making learning challenging via naive binary classifiers but effectively trainable with our proposed method. We quantitatively evaluated the performance using elaborately simulated tt̄bb̄ process data to compare the performance of the proposed method with that of binary classification. Furthermore, the proposed surrogate loss can also be applied to survival analysis. The Concordance index (c-index) metric, used as a criterion of model performance in survival analysis, constructs test data into pairs and assesses the ability to identify the data with longer survival times in each pair. We consider this metric similar to the matching efficiency; hence, survival analysis can be cast as an identifying problem similar to the initial issue in the first paragraph. Therefore, we propose a method with surrogate loss to directly maximize the c-index by reconstructing the entire data into pairs, as used in this metric. The proposed learning method with surrogate loss enables us to utilize right-censored data that have incomplete target values and are unusable for training data in the classic machine learning approach. We used three real-world datasets to evaluate the efficiencies of the data reconstruction strategies. The proposed method achieved competitive performance with partial likelihood, rank methods, and Wasserstein metric, proven effective in survival analysis, and showed no statistically significant difference.