There's a fantastic article written by James Parton in the June 1867 issue of The Atlantic Monthly entitled The City Of St. Louis, telling of a trip the author made to St. Louis in that year. The author manages to be both amazingly condescending to the people of the city while at the same time being highly impressed with all that he saw. He went so far as to proclaim that St. Louis would one day be the capitol of the United States.
Parton states that "(St. Louis) has a character all its own, to which many elements have contributed, and which many influences have modified. The ball-clubs, playing in the fields on Sunday afternoons, the billiard rooms open on Sunday, the great number of assemblies, balls and parties, the existence of five elegant and expensively sustained theatres in a town of two hundred and twenty thousand inhabitants, the closing of all the stores by sunset in winter, and before sunset in summer, and an indefinable something in the tone and air of the people, notify the stranger that he is in a place which was not the work exclusively of the Puritan, nor even of the Protestant." So I guess you can blame Sunday baseball on all of us evil, lazy Papists here in St. Louis where, the author writes disdainfully, "the civilization is...essentially Catholic."
Parton goes on to write about the influence of African-Americans on St. Louis. "As the chief city of a State that shared, and deliberately chose to share, the curse of slavery, it has much of the languor and carelessness induced by the habit of being served by slaves. The negro, too, has imparted, his accent to the tongue of the city..." Obviously suffering under the sinister influence of the Catholics of the city, even "the imitative negroes turn out on Sundays and play matches of base-ball in costume." Parton's article is the earliest reference that I've found so far of African-Americans playing baseball in St. Louis.
Note: The above picture of the St. Louis riverfront was taken from the cupola of the Old Courthouse sometime in the 1870's. The steeple of the Basilica of St. Louis, King (also known as the Old Cathedral) is visible in the picture.