Sports referees do, on occasion, swallow their whistles, but the term “swallowed his/her whistle” is an idiom that means a referee is not going to blow the whistle to call infractions. “Then in the second half, the players started getting kicked around and the referees seemed to have swallowed their whistles” was cited in 1954.
 
The term began to be used frequently starting in the 1960s. “One ref swallowed his whistle” was cited in 1962. “Referee Lloyd Gllmour apparently either swallowed his whistle or was letting the teams set the style of play, with both going at it full tilt” was cited in 1963. “The refs swallowed their whistles and never called a foul” was cited in 1964.
 
 
9 December 1954, Daily News-Miner (Fairbanks, AK), “274th Wallops 3rd Battalion” by Dan Golberg, pg. 12, col. 4:
Then in the second half, the players started getting kicked around and the referees seemed to have swallowed their whistles. The second half had all the attributes of football game.
   
18 March 1962, Beatrice (NE) Daily Sun, “Board turns thumbs down on Tech’s protest of state finals” by Al Riddington, pg. 12, col. 3:
Messer then maintained that “One ref swallowed his whistle and the other made it a grudge battle between himself and me.”
 
4 April 1963, Star-News, (Pasadena, CA), “Can Blades Do It?” by Don Pickard, pg. 26, col. 6:
Although the game was a hard-played, rough checking tilt, there was not one penalty called. Referee Lloyd Gllmour apparently either swallowed his whistle or was letting the teams set the style of play, with both going at it full tilt.
 
6 December 1964, Sunday Times Advertiser (Trenton, NJ),  “How College Recruiters Operate” by Jules P. Harlicka, pt. 4, pg. 4, col. 5:
The kid and his coach complained more than once about the unnecessary roughness, but the refs swallowed their whistles and never called a foul.
 
31 January 1968, Brazil (IN) Daily Times, “Staunton Comes On Strong, Nut Loses, 61-55,” pg. 11, col. 8:
“I was whistled for a couple of technical fouls, not because I protested the calls, but because I protested what the referees did not call. I think they must have swallowed their whistles a couple of times.”
(Yellow Jacket coach Sam Wiley.—ed.)
 
19 January 1973, Richmond (VA) Times-Dispatch, “Kings Succeed Against Robins” by Jerry Lindquist, pg. C-4, col. 3:
Referee Peter Mofatt swallowed his whistle, especially in the third period, and failed to call several obvious infractions.
 
27 January 1975, Register-Republic (Rockford, IL), “Hull drought ends as Hawks win, 3-2” (AP), pg. B5, col. 4:
“The play is getting the power play and we didn’t get many. As far as I’m concerned Wicks could have swallowed his whistle.”
(California Coach Bill McCreary on referee Ron Wicks.—ed.)
 
Google Books
Detroit Red Wings:
The Illustrated History
By Richard Bak
Dallas, TX: Taylor Publishing Company
1997
Pg. 223:
The referees had, in time-honored fashion, swallowed their whistles by now, leaving it up to the players to trip, whack, and wrestle each other until a decision was reached.
 
Google Books
When Nothing Else Matters:
Michael Jordan’s Last Comeback
By Michael Leahy
New York, NY: Simon & Schuster
2004
Pg. 280:
Afterward, he said the referees had swallowed their whistles in not calling a foul on the controversial ball strip.
 
Google Books
Why Soccer Matters
By Pelé with Brian Winter
New York, NY: Celebra
2014
Pg. ?:
And in 1966, it was as if the referees had swallowed their whistles.
 
In that first game, against Bulgaria, we might as well have been playing in a dark alley back in Baurú, with knives and clubs and not a referee (or even a concerned parent) in sight..