“Don’t change horses in the middle of the stream” (or, “don’t swap horses in midstream”) means that once something has already started, it’s too late to make changes. In politics, it means that once a candidate has been chosen, there comes a time when it is not possible to “swap horses” for a better candidate. President Lincoln has been credited for saying this at least twice: once about changing generals in the Union army (George B. McClellan would be replaced in 1862) and once about changing presidents during a war (credited to a speech before the National Union League in 1864).
   
The phrase “no time to swap horses” was well known before Lincoln became president. A joke about an Irishman uttering this line dates in print to at least 1840.
 
       
The Free Dictionary
change horses in midstream and change horses in the middle of the stream
Fig. to make major changes in an activity that has already begun; to choose someone or something else after it is too late. (Alludes to someone trying to move from one horse to another while crossing a stream.)
 
The Phrase Finder
Don’t change horses in midstream
Meaning
Don’t change your leader or your basic position when part-way through a campaign or a project .
Origin
From an 1864 speech by Abraham Lincoln, in reply to Delegation from the National Union League who were urging him to be their presidential candidate. ‘An old Dutch farmer, who remarked to a companion once that it was not best to swap horses when crossing streams.”
 
(Oxford English Dictionary)
to change (swap) horses in midstream (while crossing a stream): to change one’s ideas, plans, etc., in the middle of a project, progress, etc.;
1864 A. Lincoln in Compl. Wks. (1894) II. 531,  I do not allow myself to suppose that either the Convention or the League have concluded to decide that I am either the greatest or best man in America, but rather they have concluded that it is not best to swap horses while crossing the river, and have further concluded that I am not so poor a horse that they might not make a botch of it in trying to swap.
     
19 February 1840, New-Hampshire Sentinel (NH), pg. 1, col. 6:
A Mr. Hamer was very instrumental in bringing the meeting to his mind, by making a short speech, in the course of which he introduced the following anecdote:
 
“An Irishman, (said Mr. Hamer) in crossing a river in a boat, with his mare and colt, was thrown into the river, and clung to the colt’s tail. The colt showed signs of exhaustion, and a man on shore told him to leave the colt and cling to the mare’s tail. ‘Och! faith honey! this is no time to swap horses,’ was his reply.”
Albany Advertiser.
 
29 May 1852, Daily National Intelligencer (Washington, DC):
Let me say to my Democratic friends that this is no time to swap horses; ...
 
Google Books 
July 1859, Yankee Notions, pg. 202, col. 1:
The man who was crossing the river, and who was thrown from the boat, with a large horse and a small pony—was emphatically “quick-witted.” He seized upon the pony’s tail, that being nearest to him, for he could not swim a yard.
 
Some one on the shore cried out, “Catch hold of the tail of the big horse!”
 
“No, no,” he answered, “this isn’t exactly the time to swap horses.”
 
21 October 1860, New York (NY) Herald:
It is too late to swap horses for the Presidential race, but A. G. Brown, for President, and some New Yorkers say, Mr. Dickinson, are proposed by a Southwestern man as nads that would win.
   
8 October 1861, St. Albans (VT) Daily Messenger:
In the words of President Lincoln, “it is no time to swap horses while you are swimming a river.”
 
Google Books
26 October 1861, The Saturday Review, pg. 420, col. 2:
As the PRESIDENT is said to have remarked, with characteristic elegance of language, “it’s not the time to swap horses when you are swimming a river.”
 
1862, Vanity Fair, pg. 50:
Cabinet Conversation Pieces, No. 4
(...)
PRESIDENT:  Not yet, Sir, not exactly yet. What I said to the Boston Delegation was: “Gentlemen; it ain’t no time to swap horses when we are crossing a torrent.”
SEC OF STATE:  How grand that image! No Sheik of the swift nomades of the Eastern deserts ever let fall a finer metaphor than that “ere.”
PRESIDENT:  Hear me through, Sir, if you please.  Says I, “It ain’t no time to swap horses when we are saddle-bags deep in a torrent…”
   
OCLC WorldCat record
Never swap Horses when you’re crossing a Stream. [Song.] Words by Harold Robe.
Author: Jesse M Winne
Publisher: New York : Leo. Feist, [1916]
Edition/Format:  Musical score
 
OCLC WorldCat record
Don’t change horses; words and music by Al Hoffman, Milton Drake [and] Jerry Livingston.
Author: Al Hoffman; Milton Drake; Jerry Livingston
Publisher: New York, Drake, Hoffman, Livingston [©1944]
Edition/Format:  Book : English
   
OCLC WorldCat record
Souvenirs
Author: Dan Fogelberg
Publisher: New York : Epic : Distributed by CBS Records, 1974.
Edition/Format:  Music CD : CD audio : Popular music : English
(...)
3. Changing Horses
 
Google Books
Safire’s Political Dictionary
By William Safire
New York, NY: Oxford University Press
2008
Pg. 193:
don’t change horses A metaphor urging voters to continue an Administration during a period of crisis.
 
The figure of speech was Lincoln’s, spoken at the Republican convention of 1864 to a delegation from the Union League Club which had hailed the action of the convention in nominating Lincoln. Reported his secretaries, Nicolai and Hay: “The President answered them more informally, saying that he did not allow himself to suppose that either the Convention or the League had concluded that he was either the greatest or the best man in America, but rather that they had decided it was not best to ‘swap horses while crossing the river.’”
 
Etymologists Sperber and Trittschuh traced the origin to an 1846 newspaper: “There is a story of an Irishman who was crossing a stream with mare and colt when finding it deeper than he expected, and falling off the old mare, he seized the colt’s tail to aid him in reaching the shore. Some persons on the bank called to him, advising him to take hold of the mare’s tail, as she was the ablest to bring him out. His reply was, that it was a very unseasonable time for swapping horses.”