The command which we tried was below
BACKUP DATABASE master TO URL = 'https://sqldbprodbackups.blob.core.windows.net/daily/master.bak' WITH CREDENTIAL = 'BackupCredential' GO
Here was the error which we were seeing.
Msg 3292, Level 16, State 6, Line 1
A failure occurred while attempting to execute Backup or Restore with a URL device specified. Consult the operating system error log for details.
Msg 3013, Level 16, State 1, Line 1
BACKUP DATABASE is terminating abnormally.
We have checked and verified that same command works from another server and backup works fine. That means the issue was not with the storage account. I found below MSDN link to troubleshoot the issue.
As per above, I enabled trace flag 3051 to get more detailed messages.
DBCC TRACEON (3051,3605,-1); GO
After this, I ran the backup command again and here is the information I received in the ERRORLOG file
2018-07-04 20:52:20.83 spid65 DBCC TRACEON 3051, server process ID (SPID) 65. This is an informational message only; no user action is required.
2018-07-04 20:52:20.83 spid65 DBCC TRACEON 3605, server process ID (SPID) 65. This is an informational message only; no user action is required.
2018-07-04 20:52:23.37 spid65 VDI: “C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Binn\BackupToUrl.exe” “b” “p” “xxxx” “yyyy” “zzzz” “NOFORMAT” “4D005300530051004C00530045005200560045005200” “C:\Program Files\Microsoft SQL Server\MSSQL14.MSSQLSERVER\MSSQL\Log” “DB” “6D0061007300740065007200” “TRACE”
2018-07-04 20:52:23.37 spid65 BackupToUrl: couldn’t load process Error Code: 80070002
2018-07-04 20:52:23.37 Backup Error: 3041, Severity: 16, State: 1.
2018-07-04 20:52:23.37 Backup BACKUP failed to complete the command BACKUP DATABASE master. Check the backup application log for detailed messages.
I have truncated messages to fit into the blog, instead of xxx, yyy and zzz there were long strings. Did we see anything interesting? Below is an interesting message.
BackupToUrl: couldn’t load process Error Code: 80070002
You can use my earlier blog to convert above code to a meaningful and human-readable error message.
As per code, it means “The system cannot find the file specified.”. When I checked BINN folder we found that someone renamed BackupToUrl.exe to BackupToUrl.exe.dll
Once we renamed the file back to original name, the backup started working fine.
Here is the command to turn off the trace flag.
DBCC TRACEOFF (3051,3605,-1); GO
Have you used any other trace flag for troubleshooting backup/restore issue?
Reference: Pinal Dave (http://blog.SQLAuthority.com)