The Maryland Court of Appeals rejected a new trial for the Serial subject, whose case has been turned into an HBO documentary airing Sunday.