SharePoint Content databases move from one farm to another – data refresh process across farms


            Today we will see how we can get all the content (including Site Collections and Content Databases) from one SharePoint Farm to another farm. This also including content from Project Server Databases.

          While we are planning to upgrade SharePoint 2010 to 2013, we need to replicate the upgrade process in staging farm  to be as close to production farm. The main goal here is to move all the current content and from production farm to a staging farm where both the SharePoint farms share a similar architecture in terms of servers, service applications and web applications.

Note: project server db refresh process will follow in next post 🙂

The High level Plan:

  1. Gather all the web applications and their content databases.
  2. Get all the content databases from TST/Stagning env. where you want to detach all the databaess.
  3. Detach all the databases from the farm using a simple command or script (check below)
  4. Once verified all the databases have been detatched and remove them from DB servers.
  5. Backup and restore databases from PROD db server to this staging farm.(make sure the db owner is changed to farm service account)
  6. Mount the databases restored from prod and set the necessary max and warning site count.
  7. Restore any custom databases which are being used for any third party tools or custom solutions from PRD to Stage farm
  8. Clear all the SharePoint Timer cache.
  9. Reset all the search index data for all the search applications and once attached all the databases fire a full search.
  10. If you have blog cache implemented then you need to refresh the cache (delete and let it rebuild again)
  11. Restore “emailenabledlists” table from sharepoint_config database (what ever is the config db in prod ) to stage (optional: this isonly required if you need the email enabled lists and email details of the lists/doc. libraries/discussion forums you want to restore)
The remaining challenges for the staging farm to be identical was to migrate the managed metadata database from prd farm and 
Now lets see the basic PowerShell cmdlets to be executed for the above steps:
Web application – site collections and content databases can be gathered: (execute for each web application)
PowerShell shell:>Get-spcontentdatabase  -webapplication http://webapplication | select name  > c:webapplicationname-contentdatabase.txt
Note: In case if we use more than one SQL DB instances, we need to make sure content databases are restored in respective DB instances as per the farm architecture.
UnMount Databases – Simple PowerShell script – copy all your databases we captured in the above step to one file:
$dblist = get-content c:stg-detatchdb.txt
foreach($db in $dblist)
{
      dismount-spcontentdatabase -identity $db -Confirm:$FALSE
}
Attached databases in Stg farm:
$dblist = get-content c:stg-attachdb.txt
foreach($db in $dblist)
{
    
Mount-SPContentDatabase -WebApplication http://webappname -Name $db -Confirm:$FALSE -DatabaseServer “dbsreverinstance” -MaxSiteCount -WarningSiteCount
# your can use the below command to again set the max and warning site count
#Set-SPContentDatabase -identity $db -Confirm:$FALSE -MaxSiteCount 1 -WarningSiteCount 0
}
CLEAR SHAREPOINT TIMER CACHE:
You can clear the timer cache either manually on each server in the following way 
Either you can clear the cache using a simple script:
Add-PSSnapin Microsoft.SharePoint.PowerShell -erroraction SilentlyContinue
$servers = getspserver | ? { $_.Role eq “Application” }
$configDb = Get-SPDatabase | ? {$_.TypeName -eq “Configuration Database”}
$guid = $configDb.Id
foreach($spserv in $servers)
{
set-service -computername $spserv -Name sptimerv4 -status stopped
sleep 5
Remove-Item “\$spservc$ProgramDataMicrosoftSharePointConfig$guid*.xml”
Set-Content “\$spservc$ProgramDataMicrosoftSharePointConfig$guidcache.ini” “1”
sleep 5
set-service -computername $spserv -Name sptimerv4 -status Running
}
sleep 300
DISABLE BLOB CACHE SCRIPT:

Add-PSSnapin Microsoft.SharePoint.PowerShell -erroraction SilentlyContinue
$WebAppServices=Get-SPWebApplication
foreach($webApp in $WebAppServices)
{  
     Write-Host $webApp
     Disable-SPBlobCache($webApp);
}
Remove-PsSnapin Microsoft.SharePoint.PowerShell
This completes the content db restore from one farm to another, I will write how to restore project server db refresh in next post.
Do write me back on your opinions and if we can improve on this process and include any missing pieces.
Advertisement

Required Permissions in SharePoint Farm for executing PowerShell scripts


Before you execute any of the SharePoint Cmdlet you must be sure that you have necessary permissions to execute in the farm. Also you must have access to local WSS_ADMIN_WPG group where you are executing the cmdlet. Also you must be member of the SharePoint_Shell_Access SQL role within the databases your cmdlet calls in to interact with. Some commands also require you to be a member of the local administrator and farm administrator grooups. For ex. to create a web application, you must be a local administrator and farm administrator.

In Regards to adding users to the WSS_ADMIN_WPG group and SharePoint_Shell_Access SQL role, you must not be adding users directly to these groups rather you should use Add_SpShellAdmin cmdlet. This cmdlets accepts optional database name as parameter. If you do not pass this parameter it will add the user to the SharePoint_Shell_Access SQL role in the SharePoint configuration database only.

Above third line of code, it will add Add_SPShellAdmin SQL role in all the SharePoint databases including configuration database. Also you must be farm administrator and you must have securityadmin SQL role in order to execute this cmdlet.

  • Use Remove-SPShellAdmin to reveke access

When it comes to removing access you should use the above command to remove permissions, however it will not remove access to the user from WSS_ADMIN_WPG group on any server. You should remove this access manually. I recommend you should create “SharePoint Shell Access” AD group or groups if you want more fine grained control and grant these group Add-SPShellAdmin access there by avoiding the case users retaining the rights, where they shouldn’t.

When we get to the point where we need to execute scripts you may have to set execution policy  to allow scripts to run. You could do this using Set-ExecutionPolicy  by passing the execution policy.

  • Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope Process
While setting policy for production farms I recommend using policy scope to Process so it does not persist other PowerShell instances.

Can you move a single large site collection into multiple content databases?


Although single content databases of upto 200GB is possible in Sharepoint 2010, administering and managing such a db would be a nightmare. What would you suggest for options in a case where there is one site collection and one corresponding content database. Can you have more than one content db for that one site collection? Can you expand the table to be hosted on other sql servers/machines? Does RBS help?
Expert Comments:
Officially there is support up to 4TBs with optimization, but realistically that is difficult to support and should only be used in extreme exceptions. Technically there were no real changes made to support the additional sizes, it was just an update to guidance. I still try and work with my customers to maintain databases no larger than 75-100GBs unless absolutely necessary. I do have customers with multi-terabyte farms.
A single site collection can only use 1 Content DB. It is possible though to create solutions that use a single site collection that interacts with multiple site collections for content storage. Its transparent to the users, but splits the content across multiple sites, multiple content dbs.
RBS can also help. The big advantage there is it reduces storage within the content db. I’m not as sold on RBS though in most scenarios. There are benefits, but there are also costs and added complexity. If you have an extremely large data source that MUST be in a single site collection and content cannot be archived, then this may be a good solution.
As for multiple SQL servers, you can leverage multiple SQL servers for a single farm, but my understanding is that a single database and all supporting data/log files must be part of the same instance. Using multiple data files on separate disk partitions though is part of the guidance to increase performance on those content dbs above 200GB.

Sharepoint 2013 Service Pack 1 is coming… early next year 2014.


Service Pack 1 is coming…

Bill Baer (@williambaer) is a senior product manager on the SharePoint marketing team.

Today we broadly announced the upcoming availability of Service Pack 1 for Office, Exchange, and SharePoint early next year. 

Service Pack 1 represents a major update to SharePoint, establishing a new baseline for support, and provides customers the latest in improvements to performance, stability, and security.

Update from Microsoft Sharepoint Blog

I want to give you a quick heads up that early next year we’ll release Service Pack 1 (SP1) for the 2013 set of products including Office, SharePoint and Exchange.  SP1 will deliver performance enhancements, feature updates and improve compatibility with Windows 8.1.  

PowerShell Cmdlets to Upgrade Content Databases from SP 2010 to 2013


In this new article, I will summarize basic Powershell cmdelts we have available to upgrade SharePoint 2010 content databases to SharePoint 2013:

  • Before going into detail, you can list available cmdlets by executing the following PowerShell sentence in the SharePoint 2013 Administration Console:

Get-help *SPContentDatabase

  • After executing this sentence, you will have a list with all the cmdlets available:

  • From this list, the cmdlets we are interested in are the following: Test-SPContentDatabse, Upgrade-SPContentDatabase and Mount-SPContentDatabase.
  • Test-SPContentDatabase cmdlet allows you to identify any issue or customization you could take into account when upgrading a content database to SharePoint 2013: features installed in the SharePoint 2010 environment that should be in the SharePoint 2013 ones, language packs installed, etc. Below, you can find the general syntax for Test-SPContentDatabase.
Test-SPContentDatabase –Name -WebApplication

  • If the command execution detects issues in the content database, it will appear in the output window. For instance, a common issue is to have orphan objects in the content database.

  • Ugrade-SPContentDatabase cmdlet allows you to upgrade a content database that has some issues or upgrade it from one product build to another one. This command is used to only to resume a failed Upgrade. The syntax of this cmdlet is quite simple as you can see in the following script:
Upgrade-SPContentDatabase -Identity

  • If you execute this command in the SharePoint 2013 Administration Console, you will have to confirm that you want to start the upgrade process in the related database. If there are no problems, a warning message will be shown indicating that the database doesn’t need to be upgraded.
  • Finally, you have to execute the MountSPContentDatabase cmdlet in order to add a content database to an existing web application. Please note that only once you have fixed any existing problem found with the content database,  should you execute this command.  You are then ready to add to an existing web application. Mount-SPContentDatabase syntax is quite simple as you can see below:
Mount-SPContentDatabase -Name -WebApplication
And that’s all there is to know about upgrading from basic PowerShell cmdlets to SharePoint 2010 content databases.


Courtesy: Juan Carlos