This is indeed a fungus.  It looks like Xylario polymorpha or Dead Man's Fingers. http://www.mushroomexpert.com/xylaria_polymorpha.html  It is not generally considered to be edible.  If you do not like it shovel it up and put it into the trash.  The spores exist in the soil and when the weather conditions are appropriate, they will grow.  We doubt the presence of your neighbors vegetable garden has much to do with its growth.  It tends to grow on rotting wood.  We have noticed more of it this year than in the past, probably due to the very wet spring.  vw

Thank you. The neighbor's vegetable garden was planted over the stump and roots of a tree they cut down. Likely that some of that decaying root is on my side of the fence. Mystery solved.