“To lose one’s marbles” is to go crazy; someone who has “lost his marbles” has lost his mind. The phrase comes from the game of marbles. When one is playing “keepsies” (or “for keeps”), the winner takes all the marbles and the loser loses his/her marbles.
 
The saying developed from 1850-1900. “It sounds like the passionate ravings of a school boy who has lost his marbles at a game of ‘keeps’” was printed in the Daily Shreveport Times (Shreveport, LA) on March 28, 1876. “Mr. Gladstone ‘lost his marbles’” was printed in the Sheffield and Rotherham Independent (Sheffield, UK) on January 16, 1878.
 
The saying has nothing to do with the Elgin Marbles (or Parthenon Marbles) that were taken from Greece to England and have been displayed in the British Museum since about 1820.
     
 
Wikipedia: Marbles (toy)
A marble is a small spherical toy often made from glass, clay, steel, plastic or agate. These balls vary in size. Most commonly, they are about 1 cm (1⁄2 in) in diameter, but they may range from less than 1 mm (1⁄30 in) to over 8 cm (3 in), while some art glass marbles for display purposes are over 30 cm (12 in) wide. Marbles can be used for a variety of games called marbles.
(...)
Marble terminology
“Keepsies” (or “for keeps”): the player keeps all the marbles he or she wins.
 
(Oxford English Dictionary)
marble, n. and adj.
In plural. colloquial (orig. North American). Mental faculties; brains; common sense. Usually in to lose one’s marbles, to have (also not have) all one’s marbles, and variants.
1902   G. V. Hobart It’s up to You 64   I see-sawed back and forth between Clara J. and the smoke-holder like a man who is shy some of his marbles.
1927   Amer. Speech 2 360   Marbles, doesn’t have all his (verb phrase), mentally deficient. ‘There goes a man who doesn’t have all his marbles.’
1935   A. J. Pollock Underworld Speaks 75/1   Marbles, the brain.
     
4 July 1861, The Youth’s Companion (Boston, MA), “Loss and Gain,” pg. 106, col. 1:
It is easy to know which of those eight boys that figure in our cut this week, is the unfortunate one that has lost his marbles. Poor Tom Cram sitting, in the foreground, with his knuckles in his right eye, and the tears streaming down his cheek, betrays himself as the unlucky boy to the most casual observer.
 
11 May 1870, The Methodist Advocate (Atlanta, GA), “Playing for Keeps,” pg. 76, col. 1:
“Not for keeps,” said he, when the game begun. “I won’t play for keeps.”
 
“Oh, how mean!” cried the other boys. “He is afraid he will lose his marbles.”
     
28 March 1876, Daily Shreveport Times (Shreveport, LA), “A Remarkable Document,” pg. 2, col. 2:
It sounds like the passionate ravings of a school boy who has lost his marbles at a game of “keeps,” and wishes to charge his schoolfellows with putting up a job and cheating him.
     
16 January 1878, Sheffield and Rotherham Independent (Sheffield, UK), “Neutrality Meeting at Mexbro,” pg. 3, col. 5:
He was surprised to hear people say the present Government wanted war. Mr. Gladstone “lost his marbles,” and it was through his meddling and muddling that we were in this mess, and if Gladstone were here he would tell him so. 
 
17 September 1892, The Daily Democrat (Huntington, IN), pg. 2, col. 3:
WHEN the Herald and its gang get downed they talk about licking the editor of THE DEMOCRAT and whine around like a little boy who has lost his marbles playing “pumps for keeps.” Poor weakling.
 
21 May 1893, St. Louis (MO) Post-Dispatch, pg. 36, col. 5:
CHRIS’ KICK.
Alderman Cronin Talks About Von der Ahe’s Objection to Being Taxed.
(...)
“Yes, Von der Ahe. That guy is puttin’ up a spiel that ‘ill drive the gang crazy.”
 
“Matter with him?” replied Mr. Cronin. “Nothing—losing his marbles, that’s all.”
   
8 October 1896, The Evangelist (New York, NY), pg. 30, col. 1:
UNCOMFORTED.
Little Isabel was in tears. “What are you crying for, little one?” mamma asked.
 
“Oh, mamma,” said Isabel, “Walter lost his marbles, and when I tried to comfort him, he uncomforted me.”
     
27 January 1900, Kingman (KS) Weekly Journal, pg. 8, col. 6:
But the day after it happened he said nix and looked as glum as the boy who had lost his last marble at keeps.
 
22 September 1900, Harrisburg (PA) Star-Independent, “Panic Stricken Senator Hanna,” pg. 4, col. 2:
When the Republican chairman “lost his marbles” on the occasion of his silly speech before the Commercial McKinley club, of the same city, he revealed the fact that he is absolutely without confidence of any sort.
 
8 March 1902, The Evening Herald (Parsons, KS), pg. 4, col. 2:
The Boer delegates after their conference with President Roosevelt, must have felt a good deal like the small boy who has lost his last marble at playing “keeps.”
 
OCLC WorldCCat record
Dr. Carbles is losing his marbles! : My weird school #19
Author: Dan Gutman; Jim Paillot
Publisher: New York : HarperTrophy, ©2007.
Series: My weird school #19
Edition/Format:   Print book : Fiction : 1st Harper Trophy ed
Summary:
Tired of following the strict rules of their grumpy replacement principal, second-grader A.J. and his classmates try to find a way to bring back eccentric but lovable Principal Klutz, who was fired for bringing a wild turkey to the school’s Thanksgiving Day assembly.
     
,a href=“https://www.urbandictionary.com/define.php?term=losing%20my%20marbles”>Urban Dictionary
losing my marbles
The act or process of going crazy, usually caused by life stressors. Also used in the past tense phrase: lost my marbles. The orgin of the phrase is unknown but has been referenced numerous times in popular culuture; such as in the peter pan film Hook from the 90’s.
(...)
by yabayabadoo August 12, 2011
 
OCLC WorldCat record
GRANDPA HAS LOST HIS MARBLES.
Author: SIENNA COWING
Publisher: [Place of publication not identified], MASCOT Books INC, 2017.
Edition/Format:   Print book : English