In a recent user group meeting in my city Ahmedabad, I have found that not every user knows the difference between these two features of SQL Server. I do not blame any user for not knowing the difference between Database Mail and SQLMail as this is very confusing sometimes. I will try to explain this concept here.
Database mail is a newly introduced concept in SQL Server 2005 and it is the replacement of SQLMail of SQL Server earlier version. It has many enhancements over SQLMail. Database Mail is based on SMTP (Simple Mail Transfer Protocol) and also very fast and reliable where as SQLMail is based on MAPI (Messaging Application Programming Interface). Mail depends on Service Broker so this service must be enabled for Database Mail. It can be encrypted for additional security. SQLMail is lesser secure as it can encrypt the message as well as anybody can use SMTP to send email. Additionally, for MAPI to be enabled for SQLMail it will require Outlook to be installed. All this leads to potential security threat to database server.
In summary, I suggest if you are using SQLMail, it is the right time to upgrade to Database Mail.
If you still want to use SQLMail you will have to enable it with specific commands.
EXEC sp_configure 'SQL Mail XPs', 1 GO RECONFIGURE GO
Again, I suggest upgrading to Database Mail and start to stop using SQL Mail.
Reference: Pinal Dave (http://blog.SQLAuthority.com)