I have helped many clients in upgrade failures via my On-Demand consulting and I am amazed to see various ways which can break SQL Server upgrade. In this blog we would learn about fixing error Script level upgrade for database ‘master’ failed because of upgrade step sqlagent100_msdb_upgrade.sql
Below are few earlier blogs which have other causes of same error:
SQL SERVER – Error 15559 – Error 912 – Script Level Upgrade for Database ‘master’ Failed
As I mentioned in an earlier blog, we need to look at the exact error in ERRORLOG. In my client’s case, here is what I saw in ERRORLOG.
- Error: 8355, Severity: 16, State: 1.
- Server-level event notifications cannot be delivered. Either Service Broker is disabled in msdb, or msdb failed to start. Event notifications in other databases could be affected as well. Bring msdb online, or enable Service Broker.
- Error: 926, Severity: 14, State: 1.
- Database ‘msdb’ cannot be opened. It has been marked SUSPECT by recovery. See the SQL Server errorlog for more information.
- Error: 912, Severity: 21, State: 2.
- Script level upgrade for database ‘master’ failed because upgrade step ‘msdb110_upgrade.sql’ encountered error 926, state 1, severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
- Error: 3417, Severity: 21, State: 3.
- Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.
- SQL Server shutdown has been initiated
- SQL Trace was stopped due to server shutdown. Trace ID = ‘1’. This is an informational message only; no user action is required.
I have put line number for clarity. Message in line number 7 onwards are scary and don’t trust/follow them. Upgrade failure is in line number 5 and 6. If you read complete message in line number 6, it tells error 926 encountered. Now, line number 3 is error number 926 which means database msdb can’t be opened.
WORKAROUND/SOLUTION
As mentioned in all earlier articles, we need to start SQL Service with trace flag 902 to bypass upgrade script and fix the cause of upgrade script failures. So, here are the steps I have done.
As I mentioned earlier, first we started SQL with trace flag 902. I started SQL using trace flag 902 as below via command prompt.
NET START MSSQLSERVER /T902
For named instance, we need to use below (replace instance name based on your environment)
NET START MSSQL$INSTANCENAME /T902
Refer: SQL SERVER – 2005 – Start Stop Restart SQL Server from Command Prompt
As soon as SQL was started, we were able to connect because upgrade didn’t run. Now, we need to fix issue i.e. fix the reason of MSDB suspect database. In my client’s situation, we restore to last known good backup of MSDB database. Once restore was completed, we stopped SQL Service and started normally (without any trace flag). This time upgrade scripts ran fine, and SQL was up and running.
Have you faced any other upgrade script failures? I would be happy to write a blog, if not written already, with due credit.
Reference: Pinal Dave (https://blog.sqlauthority.com)
10 Comments. Leave new
Very good article. thank you very much for the help
The way that we fix this problem is to
A: Start SQL Server with the T902 Flag
B: Find the actual script msdb110_upgrade.sql in the Program folder for SQL Server
C: Copy the entire script into SQL Server and execute it.
D: We find the error generated – line number and go to that part of the script. Usually we have to — REM out the offending code which in our case is usually
EXEC sp_configure ‘show advanced options’ ,@advopt_old_value;
and
EXECUTE #sp_enable_component ‘Agent XPs’, @advopt_old_value out, @comp_old_value out
E: We then copy the updated script into a new query window – this part is important because if you rerun the script in the same window the entire thing fails – and run it again to make sure it works
F: We then take the updated script and replace the content of the msdb110_upgrade.sql file in the program folder
G: We then remove the T902 flag and restart the instance.
Thank you so much , you saved my production server
I performed the SP3 upgrade to 2014 SQL Server, However i was unable to start my SQL Services from Configuration manager.
I checked this article, and i run the following commands in the command prompt:
NET START MSSQLSERVER /T902 ( As my instance is default instance)
Immediately SQL Services started successfully.
I hope my case helps for some, who experience the same.
Once again thanks for this article.
Thanks & Regards,
Vivek Joginapally
SQL Server DBA
Update CU13 for MS SQL Server 2016.
Helped me with this. Thank you.
Server principal ‘##MS_PolicyEventProcessingLogin##’ has granted one or more permission(s).
Cannot recover the master database. SQL Server is unable to run. Restore master from a full
Solution:
NET START MSSQLSERVER /T902
REVOKE VIEW DEFINITION ON LOGIN::[##MS_PolicyEventProcessingLogin##] FROM public;
GO
Thanks for this solution
—Following extract from in-place upgrade log ———
Feature: Database Engine Services
Status: Failed: see logs for details
Reason for failure: An error occurred during the setup process of the feature.
Next Step: The upgrade process for SQL Server failed. Use the following information to resolve the error, and then repair your installation by using this command line: setup /action=repair /instancename=MSSQLSERVER
Component name: SQL Server Database Engine Services Instance Features
Component error code: 0x851A001A
—Following extract from sql server log ———
2020-09-02 10:27:20.63 spid25s Error: 701, Severity: 17, State: 123.
2020-09-02 10:27:20.63 spid25s There is insufficient system memory in resource pool ‘internal’ to run this query.
2020-09-02 10:27:23.63 spid5s Database ‘master’ is upgrading script ‘upgrade_ucp_cmdw_discovery.sql’ from level 184556377 to level 218107809.
2020-09-02 10:27:28.70 spid5s Error: 3417, Severity: 21, State: 3.
2020-09-02 10:27:28.70 spid5s Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup
—following steps done to resolve the issue, hope it helps …
Increase Physical memory from 16gb to 48gb
NET START MSSQLSERVER /T902
connect with ssms and changed the max memory from 8gb to 45gb
re-booted the server and the sql server service started successfully, after that i ran the repair and completed successfully.
setup /action=repair /instancename=MSSQLSERVER
In my case I got a 574 error that made master unrecoverable after attempting the SQL 2019 CU8 security upgrade.
This happened because the #MS_SSISServerCleanupJobLogin## login was not defined correctly. It didn’t exist in either msdb or SSISDB
Starting the instance with trace flag 902 gave me the chance to fix this logins permission
This included getting its schema right in msdb
USE [msdb]
GO
/****** Object: Schema [##MS_SSISServerCleanupJobLogin##] Script Date: 2/4/2021 2:28:38 PM ******/
DROP SCHEMA [##MS_SSISServerCleanupJobLogin##]
GO
/****** Object: Schema [##MS_SSISServerCleanupJobLogin##] Script Date: 2/4/2021 2:28:38 PM ******/
CREATE SCHEMA [##MS_SSISServerCleanupJobLogin##] authorization ##MS_SSISServerCleanupJobLogin##
GO
I then removed trace flag 902 and was able to start the instance without trouble and apply the failed upgrade for SQL 2019
In my case i have got below error after SQL 2019 CU10 Patch applied, not able to start sql services.
Taking SSISDB to single user mode
Setting database option SINGLE_USER to ON for database ‘SSISDB’.
Error: 3728, Severity: 16, State: 1.
‘FK_job_worker_agents_history_JobId’ is not a constraint.
Error: 3727, Severity: 16, State: 0.
Could not drop constraint. See previous errors.
Error: 912, Severity: 21, State: 2.
I tried starting sql server with trace flag and changed SSIDB to multi user mode it worked.
i still wonder what is this constraint about : ‘FK_job_worker_agents_history_JobId’ is not a constraint.Could not drop constraint.
Hi Pinal, What if you do not have an up to date backup of master db? How can I fix this issue?
Very helpful. Encountered this error for our SQL instance and this blog saved my job. Thanks man! God bless!
Upgrade script could not drop constraint FK_job_worker_agents_history_JobId because it does not exist in [SSISDB] [internal].[job_worker_agents_history] table. I have fixed this issue by recreating it. I simply scripted it in a working server and recreated in a problematic one:
USE [SSISDB]
GO
ALTER TABLE [internal].[job_worker_agents_history] WITH CHECK ADD CONSTRAINT [FK_job_worker_agents_history_JobId] FOREIGN KEY([JobId])
REFERENCES [internal].[jobs_history] ([JobId])
ON DELETE CASCADE
GO
ALTER TABLE [internal].[job_worker_agents_history] CHECK CONSTRAINT [FK_job_worker_agents_history_JobId]
GO
After that SQL upgrade has been finished successfuly.
tried all but my services still not starting sql 2022