You are running a risk of impaction. The tortoise walks through the food, dragging sand and dirt over the food, then he eats the food, sand and dirt. If he's not really, really well-hydrated, the sand builds up in the gut and blocks the 'pipes.'
Yes. It can be a skin and eye irritant and it is a big impaction risk. I have seen many xrays of sand impacted tortoises. Some people use it and get away with it. They will be along to tell you they think its fine. To my knowledge they have not seen the same xrays that I've seen. I just see no reason to risk it with so many other safer alternatives available.