First of all, you need to install the prototype in your computer. While installing the prototype, you should make sure that there is no such conflicting program already present in the computer.
Also make sure that your computer is properly protected by a firewall, in order to avoid the prototype files to be stolen over the intranet.
Evaluate different sections of the prototype one by one, and check if everything is clean - free of any bug or virus.
Go through the tutorial provided by the outsourcing company. Check whether it has catered for all the features of the software. Jot down the things which confused you in the first read, and ask the software form to re-do it.
Check your required functions one by one, and make notes of functionality of each option available in the prototype. If you do not find truly fluid functionality in any of the option, make a clear note of it and give a detailed feedback about it to the coder.
If you find any errors, blind alleys and functional problems, you should make a list of all such issues.
After you have rejected the first prototype, make sure that you communicate with the programmer on regular basis, about the progress in the development of the program. Give honest feedback about all the functions, as this will help the programmers to work well for further iterations.
Speak to the project manager and ask him about the date for the second prototype. Usually, this is included in the contract.
Check whether the software interface is user friendly or not. It should also give a professional look.