Difference of using DataSet and Lookup File Set during Lookup Operations

Dataset stores all the information in internal format of the Dataset.

Alternatively Lookup fileset also holds the same kind of information in Files.

Then the question is why is Lookup FileSet avaliable in the datastage.

There is a minor difference between DataSet and Lookup FileSet when used of Lookup.

Here goes the explanation:

Characteristic of Lookup File Stage.
When you use a Lookup File Set stage as a source for lookup data, there are special considerations about
column naming. If you have columns of the same name in both the source and lookup data sets, the
source data set column will go to the output data. If you want this column to be replaced by the column
from the lookup data source, you need to drop the source data column before you perform the lookup.

This character help in faster processing of Lookup data when the data is Huge. When a Dataset is used the Lookup data is loaded into memory and lookup is performed , in contrast when used Lookup FileSet only the Key columns are loaded into Memory for lookup and remaining columns are automatically taken from File using Inbuilt Copy Operator. This helps in processing Lookup faster when using a LookfileSet than Dataset.

However with the improvements in the Join and Sort stages , the usage of Lookup File Set is almost depleted in recent years.