We have investigated the effect of imperfectly manufactured prisms on ellipsometric measurements. We observed that some prisms showed a strong wavelength dependence in their extinction angles. To investigate this effect, assuming it's from optical activity in the quartz Rochon prism, we performed a Jones matrix calculation of the system. The result showed that the optical activity in the prism itself could not cause this kind of shift if it were well manufactured. However, we found that a mixture of ordinary and extraordinary waves could produce a wavelength-dependent shift, which might be caused by imperfect manufacturing of the prisms