Hey there!
Yes it's bad, but the only way I'd become scared is if the swing keel started banging away in a bad set of hobby horsing waves... In that situation, the whole thing could destabilize. Otherwise it shouldn't simply collapse. Make sure you sail with another boat... Someone would be there for you if the a nightmare happens.
Get it fixed at the end of the season (for sure):
What you're going to discover, is that the keel pin itself is fine, but i assure you, fine sir, the {brackets} that hold that keel pin are absolutely toast/rust-rotted/on the verge of collapse after 40-ish years of service. Like many others, I recently replaced the whole shebang. You will need someone to make you keel-pin brackets to hold the (800lb.) keel. Cap't Scott has stellar pictures that helped me through my replacement. 3M-5200 marine caulk to seal everything, then paint over everything to be positive it won't leak.
Jack the boat up, and really get your head around just how heavy 800lbs. Is. Don't play games, or you could get hurt. What worked extremely well for me, is to double cement blocks under the keel, lower back end of keel down on the blocks, leaving space for a jack at the front end of the keel. Get that jack tight, cut the pin on both sides with a saws all/metal cutting blade, then lower it down level... Here's the great part: never remove the keel from the pocket, but use round metal rod between the keel and the blocks to roll it back out of the way. Fix everything, and it would benefit you to brace your boat to prevent it from rolling while working on it. I made a round template of the boat's bottom from 2x8's, and ran a strap around it Very tightly, to prevent any rotation... While your working, keep your body on the centerline of the boat when you're inside it. Once done...hey!, you're good for another 40 years!
They're really great boats... Do the right thing, and keep it in the family.
Bill
Rotted wood: replace it, then fiberglass over it. Use Marine grade plywood.

We're just about finished with refurbishing our Chrysler 22, she's slipped, and ready for fun!
Lake Quachita, HotSprings Arkansas
Bill Williams