Coordinates: 40°44′37.8″N 73°58′54.7″W / 40.743833°N 73.981861°W
The Madison Avenue Baptist Church was first chartered in 1848 as Rose Hill Baptist Sunday School and Church, on East 30th Street between Third and Lexington Avenues in Manhattan, New York City. Rose Hill was a house church with twelve members. In 1849, Rose Hill Baptist became the Lexington Avenue Baptist Church with twenty-eight members at 154 Lexington Avenue and 30th Street in a new Lombardian Romanesque-style edifice, which is now the First Moravian Church. Prominent Baptist Jeremiah Milbank – developer of condensed milk with inventor Gail Borden – and other congregational leaders, including the Colgate family, decided to move the church east in order to avoid the falling cinders emitted by the nearby Third Avenue elevated railroad. Five lots at East 31st Street and Madison Avenue became the site of a grand new structure, built in 1858.
In 1885, following the death of Jeremiah Milbank, his wife, Elizabeth Lake Milbank, donated a memorial of stained glass windows by F. X. Zettler of Germany – sculptor of Infalbert's "Angel of the Gospel" statue – depicting the life, healing ministry, death and resurrection of Jesus. These were installed behind the pulpit.