When visitors come to New York, they usually think: Empire State Building, Central Park, Broadway. They probably rarely think: kayak to the ruins of a castle in the Hudson River. But they’re missing out. Last week, I joined a kayak tour in Cold Spring, NY, a charming Hudson Valley town about an hour outside NYC packed with antiques shops, cute restaurants, and a few great hiking trails. Our destination was Bannerman Castle, a somewhat dilapidated manse built in the early 1900s on Pollepel Island, a small circle of rock in the middle of the Hudson.

The view from my kayak, looking north up the Hudson River to Pollepel Island in the distance. Photo: Billie Cohen

The castle has an interesting and quirky history. Not surprisingly, it was the summer residence for a wealthy New York City family (the Rockefellers, FDR, and railroad tycoon Jay Gould also had mansions in this region). But the main reason the Bannerman family purchased the island in 1901 was to store all their ammunitions: guns, cannons, bayonets, and other military paraphernalia stockpiled after the Civil and Spanish American wars.

The castle’s owner, Francis Bannerman, was born in Scotland and came to Brooklyn as a toddler in 1854. He worked with his father there, collecting scrap from ships in New York Harbor to sell at auction. As an adult, Frank grew the resale business so well that he eventually started to purchase surplus government stock—including military gear from various wars. But where to put all those ammunitions while he waited for new buyers? It was too dangerous (and against the law) to keep them in the city. So when Frank’s son David spotted Pollepel Island on a canoe trip in 1901, the family bought it up and started construction of an arsenal. To give you an idea of what the Bannermans were holding, a tour guide told us that the family had so many weapons in their possession that some of the ornaments on the arsenal buildings and the nearby residence are stuck on with bayonets, set inside the walls instead of I beams.

Bannerman Castle was built as a complex to house the Bannerman family’s military resale business, which they advertised in gigantic letters on the side of the arsenal, shown here in 1905.   Photo courtesy of the Churchill / Bannerman Family/Bannerman Castle Trust

As you’d expect, this wasn’t the safest of residential complexes. And in the 1920s, an arsenal building exploded. Another fire in August 1969 did further damage, and most recently, a storm in December 2009 tumbled several walls of the main arsenal, known (in Scottish style) as Crag Inch Tower. The façade of the family’s residence is in better shape, though visitors are not allowed inside either building.

This is what the Crag Inch Tower looks like today; you can see these ruins from the mainland shore and from the Metro-North train. Photo: Billie Cohen

The Bannerman’s residence on Pollepel Island.  Photo: Billie Cohen

The ruins are currently in the care of the Bannerman Castle Trust, a nonprofit organization that has stabilized the arsenal and is trying to raise money for more preservation efforts across the island. During the spring, summer and fall, the Trust offers tours and other fun events—including a nighttime photography workshop coming up on August 23, and a five-course gourmet meal prepared by Hudson Valley chefs on September 13—and since the island is accessible by boat or kayak only, the Trust works with specific companies to organize those tours. Take note: Anyone who paddles up to the island of their own accord is prohibited from landing; you have to go with a licensed group.

My visit last week was arranged through Hudson River Expeditions. I took the Metro-North commuter rail right from Midtown Manhattan and had to walk only a block to get to the launch point, where I met our two guides and six other guests. After some instruction, we paddled for about 90 minutes from Cold Spring, then landed on the island for lunch and a guided tour (the day’s fee included the meal, bottles of water, and a donation to the Trust). We ran into a few other groups that had arrived by passenger boat, but I’d recommend the kayak route. This part of the Hudson River is beautiful; you’re not that far from the concrete and noise of the city and yet all you’ll see are low green hills lining the shore and maybe a bald eagle or two. In general, I’m a fan of kayaking as a way to see a place from a new perspective, so even if you don’t make it to Pollepel, it’s worth getting on the water to look at NYC from sea level—you can even kayak for free in the city proper at one of several coves run by the Downtown Boathouse. The best part: You don’t need to be some special athlete for these tours; all you need is a little paddling experience and a sense of adventure.