To Purge or not to Purge? – The data migration dilemma
When anticipating an upcoming Data Migration, here are a few things that you should consider in making purge decisions.
- The Time Crunch.
Depending on the application, a typical Data Migration can take at least several days to complete, if not longer.
Depending on the amount of data that one currently has, the entire process can be artificially extended. and cause you to have extra support expense and unwanted downtime.
As the amount of data that you have increased, more disk space is used and so exponentially does the number of resources required to do various tasks. This situation puts more resource pressure on your server and increases the amount of time that it takes to complete tasks. As an example, running history reports could end up taking a considerable amount of time, which may end-up reducing overall staff productivity of people are constantly waiting for something to complete. Generally, this scenario is true whether you are doing a data migration or not.
- Do I really need to migrate all of the data?
Typically, most Retailers only keep about (2) two years of detailed history. I have found many instances over the years where companies have decades of detailed history on file. However, with some industries, it may be necessary to retain more history, as examples of this might be if there were extended product warranties or high-priced items with serial numbers (e.g. Water Heaters, Pool and Spa Equipment, Wood Stoves, Tools, Farm Equipment, Electronics, etc.).
- What kind of time constraints do I have regarding the purging operations?
Normally, it is best to schedule purging related work for times when there is no one using the application software. Generally, there are at least (3) three good reasons for this as follows:
- Purging operations usually require a lot of CPU and memory usage. So if this should normally not be done during peak hours, as it can bog-down your server.
- Some historical purging operations will hang and wait if it encounters records within the selected range that are in use. This artificially extends the amount of time required to complete the purging operation.
- Smaller databases generally mean faster backup times.
- What is the general condition of the source data?
Most purging and data migration operations require that the data be in good shape, otherwise separate and extensive work may be required to accomplish complete removal of the data or the ability to migrate it.
Some examples of this kind of issue would be.
- Historical sales data that references sales reps or users that are no longer on file.
- POS ticket history that involves sales data for customer and items that are no longer on file, or that have corrupt ticket lines.
- Set-up and supporting data for features that are no longer used, such as sales prices, discount codes, and other pricing schemas.
- Old customers that haven’t purchased anything for many years. This could include customers with invalid or obsolete contact data.
- Items that are obsolete/discontinued or haven’t been used for many years.
In many cases, some of this kind of data can be cleaned up ahead of your data migration date,
If you have questions about purging, file utility usage, and data migrations, please contact the CCS Retail Systems Support Department
John