Hi and thanks for contacting Ask an Expert.
Yes, is the answer to your question. The feces and urine from the cats contain microbes and parasites that are unsanitary for vegetable growth. If the garden box is small, remove all the soil, really ALL the soil and replace it with clean soil.
Aside from the health issues for you and your family if you eat the veggies grown in this soil, the soil may be unsuitable for the plants to grow in. Heavy urine concentrations kill most plants.

This question was addressed by the Multnomah Extension Office in Oregon regarding cat feces in an asparagus patch and here is their response:

"The main problem with cat feces is the parasites that may also infect humans. Roundworms and other internal parasites can be present in the feces and end up in the soil and on your plants. Toxoplamosis is also a concern, especially for pregnant women. This organism can live in the soil for years and is known to cause serious birth defects. Children and people with compromised immune systems are also at risk from cat parasites.

Please wear gloves when working with this soil.  The situation is unfortunate but you can remedy it.  If you have further questions, please do not hesitate to contact us again.

Thank you Sheryl. I'll start changing the soil today.

Good, happy and healthy growing.