Most of math e-books you are going to find are .pdf, .ps or .djvu files. That is quite reasonable: many of the different e-books formats like ePUB, MOBI, .azw and Microsoft LIT are just a simple codification with the plain text written in the book. In contrast, mathematics books are plenty of expressions compiled with and some graphs, so a quite common source for a math e-book is a file formed from an set of images.
Kindle 2 is not able to read .ps nor .djvu files. Using Ghostscript, Ghostview and GSview you can convert your .ps files to .pdf in a simple way. Converting .djvu files to .pdf can be done using DjVuLibre with some special attention: you are going to get really huge files [even you are going to lost some pages] unless you increase the maximum image resolution between 1000 and 1200 dpi. Changing JPEG quality won’t represent any important modifications in reading quality or file size.
Finally, reading .pdf files in Kindle 2 is awful. The only way it can be done is using your display in landscape mode: this allows to have a decent font size. Cropping your .pdf file before you move it into your Kindle 2 is a really good idea [you can use Adobe Acrobat or your preferred .pdf editor].