Service Broker is message queuing for SQL Server. It is used for sending emails and through Database Mails. You can read about SQL SERVER – Difference Between Database Mail and SQLMail here. Service Broker is a feature which provides facility to SQL Server to send an asynchronous, transactional message.
So how it works is SQL Server uses Service Broker to send message from database to message queue. This message is then picked up by different processes to send this message to its destination. This separate process can be using SMTP protocol. Sending and receiving processes can be done using entirely different system or SQL Server instance. It is not required that both the services are online at the same time. Sending services or Receiving services both are not required to be online at the same time.
In summary, Services Broker follows traditional client-server architecture. Here client service initiates a conversation and receiving services takes messages and processes to send it to its destination.
Database mail is a newly introduced concept in SQL Server 2005 and it is the replacement of SQLMail of SQL Server earlier version. Database Mail has many enhancements over SQLMail. Database Mail is based on SMTP (Simple Mail Transfer Protocol) and also very fast and reliable where as SQLMail is based on MAPI (Messaging Application Programming Interface).
Reference: Pinal Dave (http://blog.SQLAuthority.com)