They usually reach full size right around 7,8 years of age. Of course, some grow faster and some grow slower. Looks like yours is one of the slower ones.
As Yvonne said, they are all a bit different, just like people.
My Jordanian Greek was about 8 years old when I got her (a guess made by the previous owner who was not the original owner). At that time she was 5 1/4 inches long and had been fed a diet consisting mostly of iceberg lettuce and lived under a normal (non UVB) heat lamp in a 10 gallon fish tank. Since she came to live with me about 3 1/2 years ago, she has been living in a 2' x 4' converted bookshelf (which still isn't big enough but all I can manage indoors) when she isn't outside (which isn't nearly often enough), under a proper UVB lamp, and eating a wide variety of tasty leafy things and cactus pads. She hasn't grown much in the last year, but in the first year after I got her and started keeping her properly she grew 3/4" and has added almost another half inch of length in the 2 1/2 years after that.
So it really depends a lot on the species, on the individual tortoise, and on their environment and diet. And even with perfect conditions, you'll still have some that grow very quickly and some that grow very slowly. Adult male Eastern Hermann's tortoises range anywhere from 3 inches to 7.5 inches in length. Yours is a bit on the smaller side for its age, but certainly not "too small" particularly if he is still growing.