It looks like initially this was caused by a caterpillar. However, the damage could also been worsened by other animals (beetles, small rodents, etc.) feeding further on the damaged area of the fruit. If you can reply with any pictures of insects or animals, we can be more specific. If possible, try covering the plants with cheesecloth or netting or screen them with hardware cloth (sturdy mesh with 1/4" square holes). Move the plants onto a deck or on top of a table that might deter any secondary feeders. 

I was working in the garden yesterday and picked a large tomato that had been resting on the ground. In the kitchen I later found this worm sitting under the tomato. There was a wormhole on the bottom. I have had very few damaged tomatoes since the first ones. Could this worm be the problem?

No, that is a worm that is in or on the ground. I believe your original holes were caused by a Tomato Hornworm. They will chew holes in your tomatoes although they also chew on leaves.  https://extension.umn.edu/yard-and-garden-insects/tomato-hornworms  I believe the prior response about secondary damage from beetles and other insects is also true. You are probably past the period of these hornworms as they are now Sphinx Moths. Watch your plants to see if anything shows up. The Sphinx Moth is a very large moth often mistaken for a hummingbird:  http://extension.colostate.edu/topic-areas/insects/hornworms-and-hummingbird-moths-5-517/