New York City has five boroughs. Each borough is part of "the city." The expression is technically no older than the city's unification (about 100 years ago), but "the city" became popular probably in the 1940s-1950s.
22 June 1969, New York Times, pg. E1:
Some people out there - when they are driving from Bensonhurst to Manhattan they talk about "going to the city" - have had it with The Big Apple.
26 August 1973, New York Times, pg. 478:
Queens Novelists and Manhattan Reviewers
By JIMMY BRESLIN
In the next few days I have a fiction book being published. "World Without End, Amen." The book begins in Queens, goes to Derry and Belfast in Northern Ireland and returns to Queens. It will be reviewed in The New York Times newspaper in Manhattan. Byron was a lot closer to Edinburgh than Queens is to Manhattan. In Queens, when people are going to Manhattan they say "I'm going to the city." Or, "I'm going to New York." People in Brooklyn say it a little differently. "I'm going over New York."
2 November 1979, New York Times, pg. A30:
Lost somewhere in America's social memory is the origin of all the putdown jokes about Brooklyn. "Where were you born?" the Army doctor asks the soldier. Soldier: "Brooklyn." Doctor: "Any other defects?" Even Brooklynites, when they go to Manhattan, say, "I'm going to the city," as if Brooklyn were out in the tulies, as they call the sticks in California.
20 July 1980, New York Times, pg. SM14:
One distinction that I believe Mr. Safire does not bring out: My mother, an unwilling Manhattan expatriate living in Brooklyn, always called the subway ride to Manhattan "going to the city." As a resident of one of the five boroughs, she was already in New York but "the city" means Manhattan. The city may sometimes be stretched to include parts of the Bronx (though Fred Allen did call the Bronx "South Yonkers"), but Brooklyn, Queens and Staten Island are beyond the pale.
22 February 1992, New York Times, pg. 26:
...changes in the City Charter that shifted power from the borough presidents to the City Council have diminished government's recognition of the sense of identity that leads people to say they live in the Bronx, and to describe visiting Manhattan as "going to the city."