the sun - Earth Rotation and Orbit Geometry

The axis points in pretty much the same direction all the time, though it does wobble slightly, ranging between 22.1 degrees and 24.5 degrees every 41,000 years. (Source)

Perpendicular between two lines is a 2 dimensional property, so, while the axis is never "straight up" so to speak, there is a 90 degree angle between the axis and the line between the center of the sun and the center of the earth during the spring and fall equinox. Those are the only times.