"Wise guy" appears to have first been used in Chicago in the 1890s, but it soon spread to New York City.
(Oxford English Dictionary)
wise guy (colloq., orig. U.S.): an experienced or knowledgeable man; usu. ironic or derog., a know-all, a wiseacre; someone who makes sarcastic or annoying remarks; also (with reversal of meaning), someone easily duped; also attrib.
1896 ADE Artie xvi. 150 He was the wise guy and I was the soft mark. 1903 H. HAPGOOD Autobiogr. of Thief iv. 82 When these Rufus's up the State get a Yorker or a wise guy, they'll strip him down to his socks. 1910 W. M. RAINE Bucky O'Connor ii. 28 You're wise guys, gents, both of yez.
15 March 1891, Chicago Daily Tribune, pg. 36:
Why didn't he step in? He was too wise a guy.
25 June 1894, Chicago Daily Tribune, pg. 11:
"In our district, which has too much territory in it," said he. "we sometimes have the divil's own time arousin' the bhoys to enthusiasm. The wise guys who makes out the list of diligates only gives us wan."
(John J. McKenna, leader of the Republican forces in the Fourth Primary District of the Twenty-eigth Ward, otherwise known as Brighton Hill - ed.)
20 January 1895, Washington Post, pg. 6:
From the Chicago Record.
Every other person in the world is a "guy" - perhaps a "wise guy," s"smooth guy," or "funny guy." The "pool shark" has grown into the belief that he is a man about town, just because he can bank a ball.
4 September 1898, New York TImes, pg. 6:
KEITH'S UNION SQUARE THEATRE. - (...) Hayes and Lytton will introduce a new sketch by George M. Cohen, entitled "A Wise Guy."