HR Data Migration and Data Cleansing

Great HR systems can transform an organisation's ability to respond quickly to environmental changes and plan for the future. But databases are nothing without data.

People data is the lifeblood of an organisation. Accurate data is critical for employee payroll, resource, scheduling, growth planning, downsizing and performance management.

• Tip 1 - Set your targets

  Aiming for 100% accuracy from large and complex data sources may, quite simply, be unachievable. Decide early on which parts of your data require the most attention to detail.

  A pragmatic approach to data quality ensures that the effort focuses on critical areas. Consider adopting a "best endeavours" strategy for less critical data.

  Agree with your stakeholders on how you will measure success. And the quality thresholds that you will aim for. Agree on the data to migrate - and the data not to migrate. Phase two is often a good option for less critical data.

• Tip 2 - Understand your data sources

  At a high level, your project might be "simply" moving data from a legacy data source to the new target platform. But even this isn't straightforward. Different platforms will organise their data in different ways. This generates gaps that need to be filled and questions that need to be answered.

  New systems might enforce rules that the old system didn't apply. Or often, the history of the old data might have caused bypassing of rules – for example, if data had to be corrected by hand.

  Strictly enforcing rules can generate problems – even when those rules seem completely reasonable. For example, every employee must have an employment start date, but what if this isn't known? This can be particularly troublesome when migrating historical data (for example for compliance reasons) when missing data can't be readily obtained.

  An early understanding of the data sources can help understand the types of issues that the project will need to address.

  It is also worth considering the impact of combining multiple data sets into a single new platform. HR transformation programmes often have to manage multiple sources for the same data – for example, separate core HR, performance, talent acquisition, payroll etc. Multiple systems with the “same” data rarely match exactly. Often the discrepancies are for good reasons (including where data changes over time), but the differences must be considered.

• Tip 3 - Ownership

  Every data migration project will create awkward questions. When these questions go unanswered, delays occur that are difficult to manage. It is crucial that the business teams support the technical people on the project rather than being left to make impossible choices. Knowing who is responsible for finding answers and making decisions can avoid problems going around in circles.

  Identify data owners (or stewards) that are ultimately responsible for guiding the choices for the technical team to implement.

  Establishing broad principles can help people make informed decisions. Data isn't usually thought of as a philosophical problem. But guiding principles will improve consistency and accelerate progress.

• Tip 4 - Analyse your data

  A review of your data early on in the project will help identify key challenges and inform the timescales. Data analysis provides a baseline for progress metrics during the project.

  Understanding the nature of the data to be migrated helps define the scope of the data to be migrated. Analysis can also help to uncover the most critical data quality issues.

  Early analysis can form the basis for ongoing project reporting and validation. Running the same analysis on the target platform helps verify the migration and can form the basis for reconciliation.

• Tip 5 - Separate the steps

  Transformation projects have a habit of creating complexity. Sometimes for good reason, the project scope is extended in an attempt to increase the perceived business value. Improving the return on investment is always welcome, but complexity must be managed, and delivery milestones must be achievable.

  Data transformation projects often become more than simple migrations. When people start looking carefully at their data, problems are usually discovered. Even though these problems may have lay dormant for years, managers may decide that they have to be fixed during the transformation project.

  Importantly, combining different elements of data transformation work can make it difficult to verify the accuracy of the changes. If data was changed to correct mistakes, how do you prove that the data was mapped correctly?

  A clear separation of the different steps provides definite milestones, clear responsibilities and better project management. For example, data cleansing work can be more effectively undertaken in the old system before migration, or in the new system after migration. Cleansing and migration in a single step can be difficult to implement reliably.

• Tip 6 - Automate

  Automation is one of the secrets to a successful data migration project. Transformations should be easily repeatable. Automating complex data migration takes more effort up-front, but this investment invariably pays dividends at many points.

  The automation can form the basis of the initial analysis and inform the project scope.

  Test runs (or "mock" migrations) can be used against a small subset of the data. In the best data migration projects, sample data can be used to develop and test the migration scripts. This helps comply with GDPR requirements for data protection. Crucially, a good set of sample data will contain various examples of data that might prove problematic. Sample datasets can speed up the debugging and fixing of mistakes in the transformation scripts.

  Automation pipelines can be run repeatably against different source and destination environments (such as dev, test and production). This repeatability mitigates the ever-changing nature of people data. Automation simplifies the eventual cutover and go-live planning.