Abstract: The study of optical encryption algorithm has become an important part of optics research today. Double random phase encoding (DRPE) in 4F system and in Fresnel domain are two typical cryptosystems. In spite of the fact that these two cryptosystems are both linear cryptosystems, the attacks on both systems are usually based on iterative phase retrieve algorithm. These crack algorithms mainly focus on the decipher of the key in the cryptosystem and seldom aim to construct the linear scrambling mapping the whole system stands for. This because one essential nature of these cryptosystems has been neglected for long time, the unitarity. With the condition of unitarity,optical attack theory can be independent from Kerckhoff' s rule and the public knowledge of the cryptosystem become irrelevant to attack algorithm. The complexity of the encryption system will not have any influence on the computation amount and crack result which is not the case in other attack algorithms. Thusly, we proposed a new method of subspace projection optical chosen-plaintext attack based on the linear and unitary nature of DPRE system, and it should fit for any linear unitary optical cryptosystems have been proposed and about to be proposed. We bring Dirac state vector and its representation linear algebra theory into our work for convenience as the Dirac state vector can stands for optical wavefront equally.

Key words: optical encryption, chosen-plaintext attack, double random phase encoding, subspace projection, quantum representation theory, unitary optical system