Abstract: Subgroup analysis of heterogeneous groups is a crucial step in the development of individualized treatment and personalized marketing strategies. Regression-based approaches are one of the main schools of subgroup analysis, a paradigm that divides predictor variables into two parts with heterogeneous and homogeneous effects and divides the sample into subgroups based on the heterogeneous effects. However, most of the existing regression-based subgroup analysis methods have two major limitations: First, they still consider the sample homogeneous within subgroups and do not fully consider individual effects; Second, the common contamination phenomenon of homogeneous effect variables is not taken into account, which will lead to large bias in the model results. To address these challenges, we propose a robust individualized subgroup analysis. We use a multidirectional separation penalty function to achieve individualized effects analysis for the heterogeneous part of the model and use γ-divergence to obtain robust estimates for the contaminated homogeneous part. We also propose an efficient alternating iterative two-step algorithm, combining coordinate descent and alternating direction method of multipliers (ADMM) techniques to implement this process. Our proposed method is further illustrated by simulation studies and analysis of a skin cutaneous melanoma dataset.

Key words: subgroup analysis, multidirectional separation penalty, robust regression, variable selection