AMSTeX is a TeX macro package, originally written by Michael Spivak for the American Mathematical Society (AMS) during 1983-1985. It is described in "The Joy of TeX" by Michael D. Spivak (second edition, AMS, 1990, ISBN 0-821-82997-1). It is based on Plain TeX, and provides many features for producing more professional-looking maths formulas with less burden on authors. It pays attention to the finer details of sizing and positioning that mathematical publishers care about. The aspects covered include multi-line displayed equations, equation numbering, ellipsis dots, matrices, double accents, multi-line subscripts, syntax checking (faster processing on initial error-checking TeX runs), and other things.
As LaTeX increased in popularity, authors asked to submit papers to the AMS in LaTeX, and so the AMS developed AMSLaTeX, which is a collection of LaTeX packages and classes that offer authors most of the functionality of AMSTeX.