As a general rule you can sue someone who makes a false statement about you that hurts your reputation subject to certain privileges. You should note that if the statement or reference is true or is an opinion it is not subject to a lawsuit for defamation (libel/slander). A former employer’s response to a prospective employer about a former employee’s job performance is widely viewed as subject to a qualified privilege, and this qualified privilege is re-enforced by an “immunity” statute, Tex. Lab. Code § 103.004. However, not all communications between a former and prospective employer are privileged. In Foust v. Hefner, 2014 WL 3928781 (Tex. App.—Amarillo 2014), the court held that the immunity granted by Tex. Lab. Code § 103.004 for communications between employers does not apply to statements an employer knows to be untrue. And in Shannon v. Memorial Drive Presbyterian Church, 476 S.W.3d 612 (Tex. App.— Houston [14th Dist.] 2015), the court held that a former employer’s statements to a prospective employer were not protected by immunity because statements were not related to the manner in which the former employee performed the job, and because the statements were in breach of agreement not disparage the former employee.
As you can see there is a privilege under the law for a prior employer to give a prospective employer but this is still dangerous because the privilege is not absolute and if the ex-employer knows the information is wrong or has bad motives for giving the reference then there is a claim for defamation (libel/slander).
Just a note on the definitions. Defamation is the general term and slander is a false statement made orally and libel is a false statement made in writing.