This does look like a young Sassafras (Sassafras albidum) sapling. They don't always have the tell-tale "mitten" shaped or fingered leaves, especially when young. (We see one or two on the youngest growth, though, as you mentioned.) They won't flower until older - perhaps around 8 to 10' tall or so.

If you wish to keep them (they are attractive plus beneficial to wildlife), you can try moving them early next spring; it's purportedly the safest time for this species; try March or maybe even late February if it's mild. They tend to be tap-rooted with few side roots when young, and are known to be tricky to transplant without dieback. If this occurs, don't give up hope immediately; they can re-sprout from the roots if they're vigorous enough.

Miri

Wow, thank you! I'm delighted to have sassafras. I'm going to evaluate its current location, and if it doesn't seem good for the long term I'm sure I can find another place. Thanks also for the information on transplanting it. I'm working on shrinking my lawn and replacing some privet; this should be an excellent addition. 

You're welcome!