If you provide an option to test something, make sure that it tests the right thing
The ZAPPATA crew was doing some spike testing today with different technologies, and due to the nature of the OS market, we did the tests with the tools most widely in use: Windows XP and Outlook.
The Microsoft-Way is a Wizard-Way. Unfortunately, some of the wizards need wizard like skills to operate or make sense of. One task that is automated by a wizard is the setting up of a new email account in Outlook. The friendly wizard takes you through different questions and sets up the mail account. We were trying to set up an account to a simple POP / SMTP server - but one that requires authentication. So we clicked on the correct options (“My outgoing SMTP Server requires authentication”, entered name and password) and then proceeded to click on the big Test Settings button provided. It found the POP server, it found the SMTP server, it checked of various other items as successfull and then failed the test of sending a message. Ok - wrong option, or so we thought. We went back to the advanced options. Changed settings. Tested. Fail. Changed settings. Tests. Looked at server logs.
...