What Is The Best Data Migration Approach?

Data Migration is one of the most challenging initiatives in IT Industry. These projects yield high business benefits (such as cost savings, increased productivity and improved data manageability) but at the same time they tend to involve a high level of risk due to the volume and critical/complexity of the data.

In order to reduce risk and ensure that the data has been migrated and transformed, we need to implement a sound validation and QA strategy with companies like datalynx.net

There are various ETL tools available in the market for Data Migration load and validation. Organizations based on their specific needs either go for the specific ETL tool or in house framework/tool. Organizations do prefer in-house tools as this would provide more flexibility in enhancement, maintenance and are cost beneficial also.

There are projects where same resource is coding the transformation logic and performing validation on migrated data. However the best approach in my mind would be to have two different resources/teams responsible for coding and validation. Both these teams would independently prepare the artifacts after business mapping has been handed off by business. Having a second set of eyes to check the mapping code and make sure that nothing was misinterpreted by developer helps address that pitfall.

There are two aspects of Data Migration testing: – Internal Data validation and External data validation.

The Internal data validation would be to make sure that we have done data migration correctly as per the mapping requirement from business.  This still does not guarantee that target environment will flawlessly process it.

This is why running the critical business processes in target environment is absolutely important. Often the outcome of this is that there are still some issues to be resolved before we successfully signed off on Data Migration. As with all issues, the earlier in the cycle these issues are identified the less the overall costs of resolution – to that end it is better to aim to get at least some of this kind of testing in as early as possible in project life cycle.

 Validation Approach Recommended:-

Here are the best validation approaches in my view based on the experience of number of client implementation projects. This is used irrespective of whether we use ETL tools or in-house framework/tools. This will also help us in answering the first five question mentioned in problem statement section.

Internal Database Testing:-

  1. Load Balancing Validations – Load Balancing among Source system and Target system to ensure that the count of data element records are in sync.
  2. Business/Scenarios based Validations (MINUS Query approach) – Business/Scenarios based Validations to ensure that the data has been correctly migrated as per mapping requirements from business. These are based on scenarios being handled in the migration which are primarily driven by the Source to Target mapping and Functional Specifications. Here we should focus on testing every rule specified in mapping for individual fields which includes Move As-Is, Transform, Look Up, Default, Calculate etc.
  3. Integrity/Sanity Validations – Integrity Validations on source data to ensure that the source data are in sync with our expectations and data integrity is ensured. This ensures that mapping rules which are based on certain assumptions/constraints in source data hold true. This eventually helps in cleansing bad source data as per the need.
  4. Data plus List Reconciliations – This type of validations is needed to ensure whether the count of data element available in source tables that should be migrating after excluding all filtered rows as per business requirements and count of data element actually migrated into the specific table are in sync. This is a high level dashboard indicating the separate reasons for dropping various data elements and list of dropped records. Once this report is available, the dropped data elements can then be reconciled with the load balancing reports.
  5. Data Reports – These reports are needed as per the request from Operation team/Business team. This is used to summarize the post migrated data element records.

External Functional Testing:-

Functional testing is required to validate that the new system meets the needs of the business but invariably functional testing is conducted on simulated data. It’s entirely possible to have migrated data perfectly and to have passed all functional testing only to find that key functions on migrated data don’t work ‘well enough’. The result is a successful operation and a dead patient – this is not acceptable. So it’s essential in my view that some level of functional testing is performed on migrated data.

Why You Should Get Your Car History Checked

To run a car check simply enter the registration number and then the mileage of the car you plan to buy. From here you’ll get lots of information, but let’s focus on the crucial parts:

Outstanding Finance

This is critical. If you buy a car with outstanding finance (and the owner doesn’t settle the balance before you actually own the car) such as Hire Purchase, Lease Contract Hire or PCP (Personal Contract Hire) you will lose ownership of the car unless you pay off the outstanding balance. But in almost all cases you will simply lose the car and any money you spent buying it. There are always varying circumstances with car financing but the overarching theme is that you will lose time and money, but most likely both. If the vehicle check shows up with money being owed via a “Stocking Loan” or “Demonstration Stocking” it simply means the car is financed by an outside finance company. Basically, the dealer doesn’t own the car. It belongs to the finance company but the dealer is the seller. These kinds of loans are very common from companies like Black Horse Finance and Moto Novo. You can use a company like DAMAGEiD car app to help you determine previous accidents too. Dealers are usually given 90 days to sell the car else the finance company will take the car back and enter it through auction. Talk it over with your dealer, check their liability and get confirmation in writing that the stock loan financing will be cleared on the vehicle before you buy it. The bottom line is, don’t take risks. If the car is showing up with finance owed, whether that be a dealer, a private sale or some other arrangement, don’t buy the car until the used car check is cleared or you have a written confirmation.

Stolen Vehicles

Car checker services reveal 40 stolen cars every single day! HPI gather information from the Police National Register for each query carried out. This means that the vehicle remains the property of the individual or organisation from whom it was taken. If your car checks flag up a stolen vehicle just walk away (and call the Police.).

Written Off

This is a term for a vehicle that has been declared a total loss by an insurer following accident damage or theft. It’s true that some written off cars can be allowed back on the road provided they have undergone certain car checks. Category ‘A’ or ‘B’ write-offs must never reappear on the road, whilst ‘C’ and ‘D’ may do so following proper repair, passing an independent Autolign-approved structural examination. Unless you are willing to spend an immeasurable amount of money on a car that’s suffered accident damage it would make no sense buying one. In fact, if you were to buy an accident damaged car you’d want it to be one that has undergone all the proper independent Autolign-approved structural examinations that come from a resulting category C or D insurance total loss. This way you will know that the car is roadworthy.

4 Techniques For Testing Data Migrated From Legacy Systems

Many stake holders of the project such as business users, project managers, business analysts really care about the data conversion scripts and the quality of the conversion. Since this conversion is business entity related and matters a lot as future business/functionality depends on the data being logically equivalent to the legacy system. Using proper Data migrations specialists is important here!

1.       Start data conversion work earlier

We have found that converting data earlier in the project life cycle is very helpful even when our data model is not stable yet. This is useful in many ways

  • It helps by forcing the developers to think and account for data in the legacy database that may not be covered by specs provided by the business analysts
  • Business analysts can communicate with the business users using real data from the legacy system that the business is used to seeing and understands the data.
  • Business users when using the new system before it goes live can understand how their new system behaves with legacy data and business entities such as Customers, Products etc. Giving the business users familiarity with the system and easing their transition to the new system.
  • When its time to go-live we have converted the data so many times that its no longer a surprise, as all the bugs, data weirdness have been found and dealt with.

2.       Automated compare of data

We compare the data from the legacy database with the new application database that is being developed. This comparison can be automated using sql that creates logically equivalent objects from both databases.

  • I blogged about usingAutomated Data Compare of comparing data.
  • We could also useDiffKit which is an open source framework that lets you compare databases, excel spreadsheets, flat files or custom formats on the legacy database side with databases, excel spreadsheets, flat files or custom formats with the new databases. Or use frameworks that export the database into xml, yml etc formats such as yaml_db

3.       Dealing with duplicates

In legacy databases, as the system gets used over many years some business entities may get duplicated and during data conversion will get merged/collapsed into a single entity. So while comparing data we should remember to do UNIQUE or DISTINCT on the SQL we use to compare the data. In some cases we may end up normalizing some of the data and end up with multiple rows for something that was represented using one row in the legacy database.

4.       Dealing with magic values

Systems use magic values to represent data or state of certain business process, such as NULL0N/ANil and other strings. When converting we cannot convert these values as-is, we need to understand what do they mean, are these values being converted to mean something else on the application front end? are we using the same logic to convert the data?

6 Reasons Product Development Is Good For Your Company

What drives established product engineering company to spend valuable resources (time, money, human capital, etc.) on new product development. Here are six reasons to invest in new product development.

Competitive Advantage:

Staying ahead of the competition should always be front of mind for existing companies.  New products give you a competitive advantage over your competition.  It might be new sales, more shelf space, consumer impressions or other advantages. If you are the market leader, you want to keep your competitive advantage, if you are not the market leader – you need a competitive advantage. New products allow you to focus on taking a competitive leadership position. There’s leadership position in the nut category is clear and their continued focus on new products keeps them in front.  Their new dessert inspired mixes are definitely look mouth-watering.

Innovation Story:

Speaking of competitive leadership, there is the credibility that comes with being the category innovator.  Innovative companies are thought of as creative and ground-breaking bringing more excitement to their brand and solving consumer problems. Being first is always better than needing to flatter your competition by replicating something they did first and you wish you had thought of earlier. 

Ancillary Sales:

Every brand wants more sales and new products help. New items might not be your number one SKU driving the most sales, but they drive new sales to your brand or category and they provide a continued look at existing products consumers may have forgotten about keeping your brand front of mind. I’m sure watermelon flavored Oreo’s are not their best seller, but each new offering generates loads of consumer media attention and new purchases.

Reason to See Your Buyer & Engage Your Consumer:
Depending on your category you might see your buyer one to four times a year…maybe monthly if you’re lucky. During those meetings you probably discuss general category data including pricing, promotions and performance, but what about products.  Same story different day doesn’t create a lot of reason for the buyer to schedule an extra meeting with you or give you more business.  New products on the other hand generate new discussion and new opportunities.

Test Consumer Trends

Consumers are constantly evolving which means their tastes and preferences change with them. In recent years we’ve see probiotic foods, gluten free, and paleo emerge as just some of the important consumer food trends. Your ability to maintain an active focus on innovation and stay current on trends allows you to be ahead of the curve with emerging consumer trends and test new concepts in early stages of trends and refine them as the trend becomes mainstream. This test and refine concept allows you to see what concepts have potential and which were fleeting while still maintaining a competitive advantage.

Explore Technology

Just like consumer trends drive new products, advances in technology drive new opportunities in various food products as well, typically in the area of food packaging. Food packaging and processing advances have opened up entirely new product opportunities. Your innovation, production and marketing team should constantly be exploring new technology that will aid in new product development.

New product development doesn’t have to be costly, but it must be a priority. Just like we talked about the power of building a marketing-minded culture, it’s equally important to include innovation as part of that ideal to foster continued organizational growth.

Why Your Business Needs A Rental Printer

Are you using an old, inefficient printer simply because you can’t justify outlaying a significant amount of cash for a new one? If so, it’s worth investigating options to rent a printer as legacy devices are probably costing your business.

Like most pieces of technology, new printers from Blackbox Solutions are much more cost-effective to run than old printers. Many newer models use less toner and paper, can be programmed to reduce overall printing volumes, are engineered to break less often and feature smart software that minimise the time your staff spend manually managing documents. This reduces the overall costs of consumables and service, and delivers significant efficiency benefits for your business. As your business grows, it’s vital to upgrade your technology environment to meet changing operational requirements. While your loyal printer may have served you well in the early days, upgrading to a smarter solution can streamline the way you work, helping you to cut costs and get ahead of competitors.

How to buy a printer without tying up cashflow

Purchasing a printer outright is rarely the most cost-effective method for buying a business printer. It ties up cash flow and makes it difficult to budget for consumables. Instead, look for rental options or pricing plans from your local IT provider. These include regular fixed payments for easier accounting and allow you to conserve cash. This means you know what you’re paying for your printing solution and when – with no nasty surprises.

  1. Rent the hardware, buy the consumables

Under a rental only contract, you pay a fixed sum each month for the printing device. Service and toner are additional costs. This option means you can purchase the device you need and set up your print environment sooner, even if it is a little more expensive, without compromising cash flow.

It’s ideal for SMEs that need to conserve cash and rely on quality printed materials for service delivery, marketing or promotions.

  1. Rent the hardware, pay a cost per page for consumables and service

This type of contract means you pay a fixed rental fee for the device each month plus a fixed cost per page to cover consumables and service (based on a minimum number of pages per month). If your business uses an expensive printer and costly toner, this type of contract can make it easier to manage and budget for consumables or servicing costs. This contract includes an extended warranty and automatic delivery of consumables, which means you don’t have to worry about remembering to order each month or being stuck without toner. The quantity of consumables depends on your business’ average monthly print volumes, which will usually be determined with your IT provider and laid out in the contract.

It’s ideal for SMEs that use lots of toner and want to avoid unexpected or uncapped consumables costs.

  1. An all-inclusive cost per page contract

This type of contract specifies an all-inclusive cost per page, based on a minimum monthly print volume. Essentially, the total cost of the device, service and toner is divided across the number of pages you print. You know exactly how much each printed page costs your business. It means you only pay for what you print (provided you meet the minimum monthly page quota). Because costs are apportioned per page, it makes it easier to compare costs when outsourcing printing and pass costs on to clients.

It’s ideal for small businesses who need to know the exact cost per printed page, especially those in client or professional services industries or those that often outsource print jobs.