Kevin Lédanois, Dion Smith, Jasper Stuyven (three days, no less), Jens Voigt, Blel Kadri somehow had a day in the jersey well into the Tour in two consecutive years, four days for Jussi Veikkanen (who apparently was quite a decent climber actually, early on, with top-50 finishes in the Giro). I think 2008 is a nice one to end it with: three days of David de la Fuente, followed by two days for Riccardo Ricco* and later seven days for eventual winner Bernard Kohl, who finished 3rd out of the blue and was disqualified due to the rather obvious doping.
* The doping king got the jersey after that epic Saunier-Duval stage where his teammates Piepoli (later DQ'ed) and Cobo finished 1-2.
Daniel Teklehaimanot had it for four days, but to be fair, it'd not be entirely right to call him a non-climber. It's just noteworthy because he's from Eritrea and that was cool.