Orthogonality of invariant vectors

Abstract

Let G be a finite group with given subgroups H and K. Let π be an irreducible complex representation of G such that its space of H-invariant vectors as well as the space of K-invariant vectors are both one dimensional. Let ^vH (resp. ^vK) denote an H-invariant (resp. K-invariant) vector of unit norm in the standard G-invariant inner product ⟨ , ⟩_π on π. Our interest is in computing the square of the absolute value of ⟨^vH,^vK⟩_π. This is the correlation constant c(π;H,K) defined by Gross. In this paper, we give a sufficient condition for ⟨^vH,^vK⟩_π to be zero and a sufficient condition for it to be non-zero (i.e., H and K are correlated with respect to π), when G=GL₂(F_q), where F_q is the finite field of q=p^f elements of odd characteristic _p, H is its split torus and K is a non-split torus. The key idea in our proof is to analyse the mod _p reduction of π. We give an explicit formula for ⟨^vH,^vK⟩_π|² modulo _p. Finally, we study the behaviour of ⟨^vH,^vK⟩_π under the Shintani base change and give a sufficient condition for ⟨^vH,^vK⟩_π to vanish for an irreducible representation π=BC(τ) of PGL₂(E), in terms of the epsilon factor of the base changing representation τ of PGL₂(F), where E/F is a finite extension of finite fields. This is reminiscent of the vanishing of L(1/2,BC(τ)), in the theory of automorphic forms, when the global root number of τ is −1.