Easily manage and customize your Zone Data Platform ETL pipeline using the Zone Data Portal. Take control of your data workflows to ensure your organization's data is accurate, up-to-date, and aligned with your needs.
Key use cases include:
- Adding new custom record types or objects from data sources
- Incorporating new custom fields from data sources
- Overseeing ETL pipeline operations, including alerts and notifications
Capabilities offered:
ETL Automation
- Configure and monitor alerting and notifications
- Manage file processing workflows
- Oversee ETL job execution and performance
Schema Management
- Add and manage schemas from data sources
- Update, remove, or modify fields within schemas
View Management
- Create and manage data views
- Update or remove existing views to keep insights relevant
With the Zone Data Portal, you’re empowered to streamline data management, resolve issues quickly, and adapt to your evolving data landscape.
ETL Management Workflows
To streamline ETL management, we’ve introduced two distinct workflows:
- Local Testing of Changes: Validate and refine updates in a controlled environment before deployment.
-
Deployment to Client Environments: Confidently push tested changes to production or client-specific environments.
ETL Change Management Details Screens
All changes are tracked using the Deployment Manager Module, ensuring visibility and control throughout the deployment process. You can access it by navigating to the “Deployments” menu and selecting the “Deployment Manager” button.
This screen will show all of the changes that have not been deployed and all of the changes that were part of a previous deployment.
Deployments
Local Testing of Changes
Our internal teams (and in the future clients and partners) do local testing before deploying these changes to a client environment. This provides the users with a quicker and risk-averse way to iterate and test in a local environment.
This process will BnL remain the same except for the Data Warehouse changes and Validation which will now leverage the Zone Data Portal. A separate article will be shared internally that covers this.
This section will focus on the Download package workflow which will allow the user to download a .dacpac file that contains ETL changes, schemas, including tables, and views.
To download the .dacpac go to “Deployments” -> “Download Packages” -> “Download Package”
This will have a pop-up that will show allowing the user to download the deployment package to their environment. There may be a pop-blocker installed in your browser that may prohibit this.
Auditing trail of downloaded packages
The “Deployments” -> “Download Packages” screen will display the details on the downloaded packages including the time it was created, the user, and the status.
Status:
Downloaded – The deployment artifact has been downloaded to the users local environment.
Deployment to Client Environments
After the Data Portal changes are confirmed the user will leverage the Zone Data Portal to push changes to the respective client production environment.
Once “Deploy All” button is pressed you will see this message
The Deployments will be in a “Queued” state
The changes will be deployed the next time there is a “Deploy Database” in the Runs like “LoadDW”.
However if the user would like to push the changes out sooner they can do so by running the manual ad hoc refresh.
Once the service has been restarted this will trigger a reload of the ETL Pipeline. The princess will be “Running”
The change will be pushed to environments post completion of the run (in case there is replicated SQL instance this step need to finish before users can see this in replicated SQL)
Auditing trail of downloaded packages
The “Deployments” -> “Download Packages” screen will display the details on the downloaded packages including the time it was created, the user, and the status.
Status:
- Queued – The deployment is queued and will begin once the current ETL process completes.
- Succeeded – Deployment completed successfully.
- Failed – Deployment failed.
Rolling back of Changes
Users can also leverage the Zone Data Portal to handle various cases of rolling back changes and then to push these changes to a production environment.
Note: Rolling back changes in a client environment will include the following steps:
- Any selected changes from any rollback will be made visible in the “Undeployed Changes” section of the “Deployments” -> “Deployments Manager” screen
- After the user has reviewed all changes they will use the “Deploy All” workflow to push these changes to the client environment
This workflow will allow a user to:
- Roll back to a previous deployment
- Roll back to a subset of changes from a previous deployment
- Additional additional changes to a previous deployment
