In running for re-election in 1969, New York City Mayor John V. Lindsay admitted that he’d made some mistakes, but that he had “the second hardest job in America,” and had done an overall good job. This became Lindsay’s 1969 campaign slogan, and he was re-elected.
 
Lindsay didn’t coin “second hardest job” (or “second toughest job”) in America. The term was known by at least 1940, during the administration of Mayor Fiorello LaGuardia, and was repeated frequently in the 1940s and 1950s.
   
     
Wikipedia: John Lindsay
John Vliet Lindsay (November 24, 1921 – December 19, 2000) was an American liberal politician who served as a member of the United States House of Representatives from 1959 to 1965 and mayor of New York City from 1966 to 1973. 

9 April 1940, Middletown (NY) Times Herald, Hugh S. Johnson syndicated colmn, pg. 5, col. 2:
One is Mayor LaGuardia of New York, who has a mind like a steel trap, can speak masterfully on any public question at a moment’s notice and has hung up the best comparative record in the country—bar none—not only as a vote-getter but as an able, honest, conservative and efficient chief executive in the second hardest job of the kind in the country.
 
21 March 1948, New York (NY) Times, “O’Dwyer Tells Why It’s Tough” by S. J. Woolf, pg. SM14:
BEING Mayor of New York is said to be “the second toughest job in the world,” the only tougher one being that of the man in the White House. New York City is a corporation which spends a billion dollars a year, employs 193,983 men and women, and provides free education for more boys and girls than the entire population of Boston or St. Louis.
All this calls for a great deal in the man who sits at the Mayor’s desk in City Hall. It also takes a great deal out of him, even in quiet times. In times of crises such as have frequently beset Mayor William O’Dwyer since he took office on Jan. 1, 1946, the multitudinous headaches of the post test human endurance to the limit.
       
3 November 1949, Cedar Rapids (Iowa) Gazette,  “Skater, Ex-Cop and First Baseman Seek N.Y. Reins,”  pg. 7, col. 1:
NEW YORK (UP)—A blue-blooded figure skater, an Irish ex-cop with the gleam of romance in his eyes and a first baseman who hasn’t had much time for fun since he was 17 are fighting for the second toughest job in American public life.
 
That job is mayor of New York, boss of its 180,000 city employes, arbiter of its endless labor disputes, chief director of its $1,000,000,000 annual budget, caretaker for a powderkeg of racial tensions and supervisor of an unbelievably complicated mechanism of modern living.
   
10 September 1950, Wisconsin State Journal (Madison, WI), pg. 1, col. 8:
NEW YORK—(UP)—New York political leaders chose candidates Saturday for a four-cornered race for mayor of the nation’s largest city, the second toughest elective job in the country. 
 
19 November 1950, Ogden (UT) Standard-Examiner, pg. 7A, cols.3-5 headline:

Wins Second Toughest Job in Nation 
   
4 November 1953, Oneonta (NY) Star, “Wagner Sweeps To Victory In New York City,” pg. 1, col. 1:
Wagner, a bland, adult collegiate type of medium size, succeeds Mayor Vincent R. Impellittieri Jan. 1 in the second toughest job in the nation—the $40,000 post as top man in New York’s City Hall. 
 
1 November 1957, New York (NY) TImes, “Advertising” by Carl Spielvogel, pg. 48:
Car cards advertising the New-York Historical Society’s exhibition, “The second hardest job in the world—Mayors of New York 1784-1957,” will appear on city subways and buses.
 
1 November 1957, New York (NY) Times, “Exhibit Traces the Mayoralty Back to 1784: Historical Collection Shows Cartoons and Portraits” by Sanka Knox, pg. 16: 
It’s said to be the second hardest job in the world, being Mayor of New York these days. The New York Historical Society is saying this in an exhibition opening today in its quarters, rich with city lore, at 170 Central Park West.
 
Ostensibly The Second Hardest Job in the World is an election year portrait salute to the long string of Mayors since 1784, the beginning of the so-called “Federal” or American period. Long walls are lined with the stylishly limned likenesses of the earlier incumbents, followed by the hirsute ones of the later Eighteen Hundreds, their high-collared successors and the moderns.
 
The present holder of “The Second Hardest Job” Mayor Wagner, at $40,000 a year, looks as easy-going as James Duane, first in the chronological line, who made less than $3,000 a year.
 
5 June 1961, New York (NY) Times, “What Wagner Didn’t Say,” pg. 30:
The people of New York City last night saw or had a chance to see Mayor Wagner’s marathon television program explaining the complex operation of the city government. They now understand better, perhaps, why it is commonly said that being Mayor of New York is the second hardest job in the political life of the United States, bowing only to the Presidency in its demands on strength and wisdom.   
 
22 June 1965, Brandon (Manitoba, Canada) Sun, pg. 4, col. 4:
The New York race will be fascinating, not just as a clash of personalities but as a test of ability for what is generally considered the second toughest job in the U.S., after the presidency.
 
The question is being asked whether, in fact, anybody can govern New York City. 
 
25 September 1966, Fresno (CA) Bee, pg. 13B, col. 2:
Was it Mayor Lindsay’s reply—the understatement of all time—to the question of a friend? The attractive head of New York City, who has probably the second hardest job in the country to being President, was asked, “How do you like your job?” He replied, with a grin, “It swings.”
 
2 May 1968, Albuquerque (NM) Tribune, “One Reason New York Escaped Holocaust: Mayor John V. Lindsay” by Richard Starnes, pg. D5, col. 1:
HE DISMISSES such talk abruptly, brushing aside the notion that visions of the Vice Presidency are dancing through his well-curried head.
 
“First off, there are two kinds of people. There are the kind who can live with that sort of job, and there are the other kind. I’m already in what most people call the second toughest job in the country, why should I settle for the third toughest?”
 
3 August 1968, New York (NY) Times, pg. 10:
Mr. Lindsay disavowed any interest in the Vice Presidency.
“People say I now have the second toughest job in the country,” he said. “I don’t see why I should be interested in the third toughest job in the country.”
 
19 January 1969, New York (NY) Times, pg. E5:
If the Mayor of New York City has the second toughest job in the country, then Mitchell I. Ginsburg probably has the third toughest—at the moment, anyway. As Administrator of the city’s Human Resources Administration, he must wrestle with an exploded welfare budget of $1.6-billion and an antipoverty program born four years ago in confusion that seems to have deteriorated into chaos.
 
2 September 1969, New York (NY) Times, “The Mayoral Contest” by Richard Reeves, pg. 32:
Mr. Lindsay’s basic early-campaign line was laid out yesterday in a speech prepared on Long Island and delivered at Rockaway Beach. That line is basically as follows:
“This is the second toughest job in the country. I’ve tried hard to make this a better city and to bring its people together. Some things have worked and some haven’t, but they’ve all given me the experience for a better second term.”
 
14 September 1969, New York (NY) Times, pg. 81, col. 1:
Despite the mellowness, Mr. Lindsay’s performance as Mayor will be tested against his own campaign slogan: “It’s the second toughest job in America.”
       
15 September 1969, New York (NY) Times, pg. 50:
Mr. Lindsay’s main campaign theme is “It’s the Second Toughest Job in America.” 
   
2 November 1969, New York (NY Times, “Here Comes the Next Mayor” by Richard Reeves, pg., SM25:
In the end, so much of it is absurd. It may be absurd to want the damn job—“It’s the second toughest job in America,” says Lindsay’s literature, and that’s no exaggeration; this is the job that Robert F. Wagner had for 12 years and people who were with him at the end say he reacted to the final daily crises with: “The hell with them. They get what they deserve.” 
   
3 November 1969, New York (NY) Times, pg. 51 ad:
The things that go wrong are what make this the second toughest job in America.
(...)
Vote for Mayor Lindsay on Liberal Column D or Independent Column F.
   
31 August 1993, New York (NY) Times, pg. B2:
Mr. (David—ed.) Garth helped re-elect John Lindsay as Mayor in 1969 with the slogan, “It’s still the second-toughest job in America.”
 
The New Yorker (January 7, 2008)
The Political Scene
Old Habits
How the Giuliani method may defeat him.
by Elizabeth Kolbert
(...)
Mayors of New York have often seen themselves as singularly suited for higher office. “It’s the second-toughest job in America!” according to John Lindsay’s famous slogan. Giuliani likes to say that having run the city “is about as good a preparation for being President as exists”—an assessment that, not surprisingly, is much the same as the one offered by Mayor Michael Bloomberg, who has flirted with running as an Independent. (In 2000, when Bill Clinton was rumored to be thinking of entering the New York mayoral race, his press secretary inverted Lindsay’s formula, observing that the President “already has the second-toughest job in American politics—why would he want the toughest?”)