Inspired by Ihara's question on the pro-$l$ outer Galois representation associated to the funcamental group of the projective line minus three points, we introduce a certain new finiteness conjecture for abelian varieties over number fields (now sometiems called the Rasmussen-Tamagawa conjecture), and present several affirmative partial results, among which some are unconditional and others are conditional (i.e. assuming GRH). This is a joint work with Christopher Rasmussen.