A novel rain or snow streak removal algorithm for a stereo video sequence is proposed in this work. We observe that stereo images or adjacent frames are highly correlated each other. Thus, we first detect candidates of rain or snow streaks by subtracting the warped another view, previous, and next frames. We then combine rain or snow streak candidates from each subtracting results by median filtering. Remaining outliers are eliminated by investigating the spatial distribution of the combined rain streak map. Finally, we replace rainy and snowy pixels with nonlocal neighbors of their related frames. Experimental results demonstrate that the proposed algorithm removes rain or snow streaks efficiently and preserves original scene structures reliably, outperforming conventional algorithms.