Have you looked at the EditText inputType attribute? You can set up a whole bunch of different input types that EditText should restrict to the user.
From the sounds of this, you are probably looking for something like:
<EditText .... android:inputType="number" ... />
source share