Fear the Walking Dead star asked to leave the show because "the beginning of this season kind of felt like the end of an era"