Anacapa Island, while surrounded by relatively calm waters when compared to those west of it, experiences dangerous fogs and Santa Ana winds. A number of vessels wrecked on or near the island although only one seemed to have been related to pilot error. The wreck of the Winfield Scott in 1853 was the most important of the Channel Islands shipwrecks.

Westervelt and MacKay of New York built the Winfield Scott in 1850 for a route between New York and New Orleans. Named for the commanding general of the U. S. Army, Mexican War hero and presidential nominee, the 225-foot steam paddlewheeler began travel between the east and west coasts when she joined the New York and San Francisco Steamship Line in 1852. She set a record for that route (via Cape Horn) of less than 49 days. She then took passengers between Panama and San Francisco, usually in overcrowded conditions. In July of 1853 the Winfield Scott was sold to the Pacific Mail Steamship Company.

Loaded with over 300 passengers and crew and a reported $1 million in gold bullion (some accounts but the figure at $2 million), the Winfield Scott departed San Francisco on December 1, 1853. The next evening Captain Simon F. Blunt chose to pass through the Santa Barbara Channel to save time, but in a dense fog crashed into Middle Anacapa at full speed at 11:00 in the evening. Apparently Blunt had turned southeast thinking he had cleared Santa Cruz or Anacapa Islands. Purser Watkins gave this account, which was published by the Alta California:

"The ship's bow struck first, staving two holes in the bow; then, in backing off, her stern struck and knocked away her rudder. Most of the passengers had gone to bed but they came on deck in alarm. They could only look out into a thick fog and see nothing of the land. After losing the rudder, the ship drifted off the island a few hundred yards and then was swept back to strike the shore again with her bow. She had already taken on much water and now sank up to her guards.

The Captain immediately sent out a boat to see where he could land the passengers. The purser reported that a little island separate from the main one was nearest and that the passengers could be placed on it for the night. The next morning the passengers, some mail, and the treasure were taken onto the main island. Captain Blunt stayed cool, provided bedding and provisions for the passengers, and moved among them unceasingly in his efforts to make them comfortable."

After being brought from the small pinnacle some 200 yards offshore to the island in the ship's boats, the large group camped on the island for up to a week. The California saw smoke from the passengers' fires and rescued the women. It returned on December 9 and, in heavy swells, removed the rest of the passengers, leaving the ship's company on the island who attempted to save mail, baggage, furniture and some machinery from the wreck. Other boats appeared for salvage opportunities, including that of Captain Horatio Gates Trussell, who used wood and two brass thresholds from the wreck in the construction of his Santa Barbara home.

On the 10th the steamer Southerner hove into sight and landed provisions for the officers and crew. Up to that time, the Winfield Scott had not been broken up by the action of the waves. When the Republic arrived the next day, however, the midship was sunk and Captain Blunt gave up all hope of saving her or getting her off the ledge. The crew went out to the ship and saved what they could before boarding the Republic for San Francisco. Some time later, the side-wheeler toppled off the ledge and sank. The valuable cargo was apparently saved although the lure of gold still attracts divers, who are prohibited by federal law from disturbing the wreck site.

In 1894 and during World War II major salvage took place on the Winfield Scott. One firm blasted the wreck in order to remove hundreds of large copper bolts and much of the iron machinery. The wreck became popular to sport divers who are now prohibited by antiquity laws from removing any artifacts. The wreck of the Winfield Scott is listed on the National Register of Historic Places.

The wreck scatter is surveyed each year by the National Park Service. The scatter is considered to be small for such a large vessel, with the bow and stern missing. The most prominent piece of wreckage is purported to be the port paddle wheel, which stands above the bottom.