Rate distortion optimization technique takes an important role in video coding, which is widely used in many video coding modules, such as macroblock mode decision, quantization optimization, etc. In this paper, rate distortion optimization coding and optimal rate control is studied jointly. First, according to the analysis on the relationship between rate and distortion, a complexity scalable rate control is proposed. The proposed rate control algorithm can achieve better coding efficiency while with accurate target bit rate; second, buffer constraint for the bit allocation of rate control is studied, and the bit allocation process is adjusted to meet the buffer constraint, so that the proposed rate control algorithm can guarantee that the buffer never overflows or underflows; finally, a constant quality rate control algorithm is developed based on the research on some subjective quality issues, such as scene change detection, macroblock classification, etc.