I have a general deployment related query, of moving artifacts from sandbox to production through sp-config APIs.
When exporting a source or rule (or other objects), a JSON file is acquired which contains created, modified and a ton of IDs (like source ID, schema ID, Identity ID, etc.) which could be tenant specific. Should those be manually removed from the files before they are imported in another tenant environment?
What are the changes expected to be done manually on exported files like Rules, Sources and Transforms?
@edmarks have you used SP config before? Do you have any advice for Sushant? I’ve never used it myself to migrate configurations between tenants, just as a backup tool.
@colin_mckibben What’s the recommended best practice when it comes to such deployments or prod-migrations? If not sp-config, is there any other process you currently follow (Apart from doing the configurations from the UI in production)?
@MVKR7T - That’s interesting. I am interested in understanding how a back/restore tool can be leveraged in deployments. Looking forward to hear more about your process.
Config Hub currently does backup/restore to the same tenant. The next feature in the roadmap will be backup/restore to a different tenant. Config Hub is a very likely candidate for performing automated migrations of tenant configuration from sandbox to prod.
Ah yes, I should have clarified that the cross tenant restore is in early preview for select customers. It will be generally available in another quarter or two. You can contact your CSM if you would like to get access to the early preview for testing purposes.
@sushant1 We’ve been using sp_config with our clients successfully for a few months now. Our approach may be different than others, but we have a custom PowerShell script that exports all objects to individual files (could be done via Postman/Curl also).
Once we have these files, we use sp_config to load them back to the new empty tenant (i.e. moving from SBX → PRD).
Order: Transforms, Sources, Identity Profiles (normally)
There is some search/replace needed for changes to object IDs, object names, etc. but to help facilitate this we use an Excel spreadsheet to track the objects across environments (i.e. some clients utilize 3 vs. 2 environments)
The challenge is that Configuration Hub will likely replace our more manual method, but this is something we could probably still document as an alternative or for situations where the standard toolset doesn’t provide for alternate use cases.
I think there’s still value in blogging about it as it can spark ideas for further automation. Your methods can probably be leveraged in a CI/CD pipeline.
Thanks @edmarks, that is insightful. The IDs that you’re replacing, is it safe to say you remove all the lines in the export JSON that start with "id":? Because I’ve tried this approach on Sources and it works, but I am not sure if it does for other object types. If you’re replacing certain IDs, is it because the export/import flow breaks if you take it out?
Specifically, when it comes to connector rules, I have always faced an error with sp-config around jswSignature. I’ve tried removing the jsw related keys from the export (these keys are usually at the bottom of the exported rule), but I still see an error.
@colin_mckibben Can you confirm if retaining the IDs, timestamps and other metadata as you mentioned earlier has no impact when moving between tenants? Because if that truly is the case, it really simplifies the entire migration process as we don’t need the overhead of pre-processing before import.
@MVKR7T Thanks for the insights around Configuration hub. While we do not see this feature yet on our client Tenants, a demo tenant we have does show this feature. I’m looking forward to see how exactly you use this end to end in a typical production deployment. Please keep us posted!
We have below options to migrate from one tenant to another or in same tenant.
VS Code with IDN Extension
SP Config
CLI
Config Hub
Custom Scripts
Postman (Manual)
Every option has it’s own pros and cons. Config Hub is still in development phase I guess. There is problem with ids sometimes. We have to see which is best for what kind of work. Will create a blog soon.
Another challenge with moving configuration is that normally you end up moving create profiles, attribute sync, enabled roles, etc. but you don’t want them active in PRD until you’re ready for this. I’m not sure if this is part of the planned changes to the configuration hub, but definitely something you can account for via some of the more manual methods.
Is it safe to say that for fresh deployments, we can ignore the various ids, created, modified fields in the json export and use the export as is for deployments.
For the most part YES! If there are no name changes needed then transforms and sources can generally be simply migrated forward. When moving the identity profiles, we generally replace the ID’s in the file for the newly created sources for consistency but that’s the minimal changes we’ve been doing for multiple client migrations.