A novel fast algorithm is suggested for a coding unit (CU) mode decision using pseudo rate-distortion based on a separated encoding structure in High Efficiency Video Coding (HEVC). A conventional HEVC encoder requires a large computational time for a CU mode prediction because prediction and transformation procedures are applied to obtain a rate-distortion cost. Hence, for the practical application of HEVC encoding, it is necessary to significantly reduce the computational time of CU mode prediction. As described in this paper, under the proposed separated encoder structure, it is possible to decide the CU prediction mode without a full processing of the prediction and transformation to obtain a rate-distortion cost based on a suitable condition. Furthermore, to construct a suitable condition to improve the encoding speed, we employ a pseudo rate-distortion estimation based on a Hadamard transformation and a simple quantization. The experimental results show that the proposed method a