T cell vaccines against Mycobacterium tuberculosis (Mtb) and other pathogens are based on the principle that memory T cells rapidly generate effector responses upon challenge, leading to pathogen clearance. Despite eliciting a robust memory CD8+ T cell response to the immunodominant Mtb antigen TB10.4 (EsxH), we find the increased frequency of TB10.4-specific CD8+ T cells conferred by vaccination to be short-lived after Mtb challenge. To compare memory and naïve CD8+ T cell function during their response to Mtb, we track their expansions using TB10.4-specific retrogenic CD8+ T cells. We find that the primary (naïve) response outnumbers the secondary (memory) response during Mtb challenge, an effect moderated by increased TCR affinity. To determine whether the expansion of polyclonal memory T cells is restrained following Mtb challenge, we used TCRβ deep sequencing to track TB10.4-specific CD8+ T cells after vaccination and subsequent challenge in intact mice. Successful memory T cells, defined by their clonal expansion after Mtb challenge, express similar CDR3β sequences suggesting TCR selection by antigen. Thus, both TCR-dependent and -independent factors affect the fitness of memory CD8+ responses. The impaired expansion of the majority of memory T cell clonotypes may explain why some TB vaccines have not provided better protection.