Sharepoint – WorkFlow Timer Job Paused


On a random daily check on CA we found workflow timer job in “Paused” state in Central Admin. Multiple of these pointing to each server in farm in paused state. Many blog posts pointing to restarting the timer service on each server and also clearing the timer cache (configuration cache) would resolve the issue.

http://www.sharepointsiren.com/2013/05/sharepoint-timer-job-stuckpaused-fix.html

http://markimarta.com/sharepoint/paused-workflows-in-sharepoint-timer-job-fix/

After the cache is cleared initially the jobs ran for a couple of instances and they were again pausing and the reason was a modified workflow resulting in a “dead end” state. Which was a mistake by the user. Each server processing workflows timer job became “stuck” due to items reaching the “dead end” state. All workflows with pauses or state changes would have ceased to be processed at this stage.

Under normal operations, a workflow is first executed in the World Wide Web Publishing Service (w3wp.exe). After a delay or pause, in which the workflow “sleeps,” it wakes up and is then run by the SharePoint Timer Service (the owstimer.exe process).

 When w3wp.exe is under excessive load, it postpones the workflow. The Timer Service is then used to the continue the process. SharePoint dictates which Workflow Timer service will run the workflow; the operation can occur on any server running the “Microsoft SharePoint Foundation Workflow Timer Service” service. However, when Nintex Workflow is installed, it only deploys the DLLs required to run workflows to WFE servers–specifically, those servers running the “Microsoft SharePoint Foundation Web Application” service.

This issue is discovered by enabling verbose logging both at CA and also at Nintex global settings and also querying for the workflow name and its site collections using Nintex NWAdmin.exe

After the workflow is modified and already the workflows which are started with the broken workflow is cleared and again restarting the timer services on servers with clearing the cache resolved the issue.

This issue is reported on SharePoint 2010 and Nintex 2010

 

 

Tags: nintex workflow, State changes error, workflow timer job paused, sharepoint timer service restart.

Nintex Workflow Migration – Data and Configuration Migration


Migration process for Nintex Workflow data from one SharePoint farm to another where the two farms are mutually exclusive and running with their own accounts and instances, like when you want to bring in PROD data to non-prod for performing content refresh for testing CU’s or other major enhancements in non-prod. Or when we have to consolidate two SP farms into one by bringing over web applications from one farm and merging into another SP farm.

The idea is to preserving the current Nintex configuration of the destination farm and trying to move only Nintex data and configuration from source farm. Nintex has a command line tool which will migrate data from source Nintex database into destination database. Here we have two options where we can move the data to existing Nintex database or we can create an additional Nintex content database and move data into newly created database. I like second option where if we have to rollback it’s easier. (SP 2010 and Nintex workflow 2010 are the context of this document)

The process can be divided into 5 Major activities:

  • DB creation
  • Migrated Data
  • Migrate configurations
  • Testing
  • (optional) Roll back

DATABASE CREATION:

How do you add a new Nintex content database: Either from Central Administration or Nintex command line tool: Nwadmin

For me: It’s easier to create the dB from CA rather than command line.workflow1

Nintex has a command line tool to migrate only the workflow data but all other configuration has to be either manually created/updated on the workflows or import the necessary dB table data into destination. We will look at that little later in the process.

Nwadmin is a command line tool that ships with Nintex Workflow 2010. It is used to perform various administration operations.

By default, the NWAdmin.exe tool is installed to the SharePoint hive, typically at the following path.

%ProgramFiles%Common FilesMicrosoft SharedWeb Server Extensions14BIN

Note: For some versions of Nintex Workflow 2010, the Nwadmin tool is installed in the installation directory, typically at the following path:

%ProgramFiles%NintexNintex Workflow 2010 (I find it here not in 14 hive)workflow2

Click to access NWAdmin_Operations_2010.pdf

Migration Process:

Pre-Migration:

Backup Source (Prod) and Destination (Non-prod) Nintex DB’s

Either bring the source database to destination sql server and assign destination farm account access to the dB or grant destination farm account access to the dB where ever it is.

Grant FARM_ADMIN account DB_owner role on Destination farm DB’s: SP_NW2010 ; SP_NW2010Source

Migration:

Consider the following points which are important while migrating the nintex data.

While migrating web applications needs to be stopped (iis) and
Timer service on all the servers to be stopped aswell. Consider gathering all the site collections where Nintex Workflows are active.

From SQL run the following query “SELECT SiteID FROM [NintexDBname].[dbo].[Storage]”
Or gathering the list of site collections in  “Central Administration : Nintex Workflow Management : Nintex Workflow Database Setup : View Database Mapping”

While web apps and timer service are off run the following command to migrate data:
nwadmin -o movedata –SiteID “GUID” –SourceDatabase “Prod Nintex DB Connection String” –TargetDatabase “Adelaide Nintex DB Connection String” –RetainSourceData

Connection String format: “Server=DBserver;Database=DBName;Integrated Security=True”workflow3

Make sure you create a sql alias so that you pass a single string in server name rather than using “serverinstancename”.

The command window output looks like below when it is success:

C:Program FilesNintexNintex Workflow 2010> nwadmin -o MoveData -SiteID “F1E52EF2-996F-46A1-A0D1-250D8970E725” -SourceDatabase “Server=TSTSP10DB-BNE;Database=SP_NW2010DB;Integrated Security=True” -TargetDatabase “Server=TSTSP10DB;Database=SP_NW2010_BNE;Integrated Security=True” -RetainSourceData

Before continuing this operation, please stop the following on each server:

SharePoint IIS web site

SharePoint 2010 Timer Service

If these services are not stopped, workflows may continue adding data, leaving the instance in an invalid state.

It is recommended that the source and target content databases are backed up before continuing.

Data will be moved for site collection ID: f1e52ef2-996f-46a1-a0d1-250d8970e725.

Restart each service to continue workflow operation.

C:Program FilesNintexNintex Workflow 2010>

Post-Migration

  • Grant db_owner access and assign WSS_Content_Applications_Pools role to all the app pool accounts of the migrated web applications on this dBworkflow4
  • Start web applications, SharePoint Timer services.
  • Login to each site collection and go to Nintex workflow and check for any errors.

Migrate Configuration:

Till here we had complete one part of the migration where we moved WF data there are other components we need to consider to move, some of them listed below.

  • Workflow constants
  • Workflow Schedules (need be recreated manually)
  • Managed Allowed Actions
  • User Defined Actions
  • Lazy Approval settings
  • Error Notifications
  • Delegation
  • EventReceivers

If you have some expertise in sql queries that would be handy in identifying differences between tables in both the DB’s

Some of the configurations I can help identify here and provide out of the box solutions and for remaining we need to use dB table move method or identify the need based on case by case.

Workflow Constants:

These will not be migrated need to migrate manually. If they are few it’s easy to add them to the workflows otherwise consider the below commands to get them updated.

Use nwadmin -o exportworkflowconstants and then import with nwadmin -o importworkflowconstants

NWAdmin.exe -o ExportWorkflowConstants -siteUrl siteUrl -outputFile pathToFile [-includeSite] [-includeSiteCollection] [-includeFarm]

NWAdmin.exe -o ImportWorkflowConstants -siteUrl siteUrl -inputFile pathToFile -handleExistingSkip|Overwrite|Abort [-includeSite] [-includeSiteCollection] [-includeFarm]

Use the nwadmin documentation I refer earlier for detailed steps.

MessageTemplate: The entries can be added back via UI (Central Admin – Nintex Workflow Management – Message Templates). Impact is that customized templates will not show in emails.

EventReceivers: Impact is if event receiver entries are missing then task related action might not work properly. The entries are added to this table at the time of Nintex Workflow features activation, if destination configuration table has already entries for sites it can be ignored. Otherwise, deactivate and re-activate the Nintex workflow features. (“Nintex Workflow 2010” & “Nintex Workflow 2010 InfoPath Forms”) on all the site collections migrated where you have Nintex Workflows active/running.

Changes in Functionality:

Please note that the send notifications email will change (if the source has a different email address) for users as only 1 address can be used per farm.

Environment From Address Reply To Address

Testing:

I suggested below configuration areas are tested in a test environment before the actual live production migration:

  • Lazy approval (not required as it isn’t configured in either farm)
  • send notifications
  • query AD
  • execute SQL
  • Web request
  • call web service
  • Any other actions that are using stored credentials to connect to other systems

Rollback Procedure:

Drop the NEW Nintex content destination DB, The source database is still not touched and still hold the original WF data.

Having said that when migrating some web applications from one farm to another you cannot bring all the Nintex configurations as the destination farm already had its own, so there is a need for good amount of testing what is needed and how it can be brought, and I hope this document will provide some of the insights into it.

The reason this is created is we could not get proper documentation on this process either online or from Nintex support.