I think that you are right and the ball barging is aggressive territorial behaviour.Lola used to push a tennis ball about but then started to walk over it and it would get stuck under him.. I tried a bigger ball but then I read somewhere here that because balls are tort shaped (to a tortoise) they are actually a bit stressed by them so barge them as they would another tort.
Lola tends to ignore them now.
I have a play tunnel from Ikea that Lola likes to walk through and lie in the middle of.
I started off with this cat play tunnel (with all attached cat toys etc removed) and he liked that so got him a bigger version.
But I don't think that's necessarily bad for your tortoise, keeps it active and behaving normally, without risk of injury.
Don't think they get too stressed as they tend to win!
And then realize it's no threat and ignore it.