What Is Microscan and Delta Versioning

Hello everybody,

Some backup software use incremental or differential backups.So, they do a full backup at the first time, then they only store the changes for next backups.Some software are more intelligent and use a monitoring solution in file or block level.
Let's say an example. You created a text file with a word of "Hello" inside the text file and saved it yesterday. The file backed up during the daily backup process. Today, you open the file, change "Hello" to "Hello1" and save it with different name. As you see here, you just add one more character to file. The second file (new file) is a version of the first file and they are almost the same. The second file is actually a delta to the original. A delta is an offset of the original file.

Now, if we use the file-compare backup method, it will backup the entire new file because it has a different file name. If we use file-level hashing method, it will backup the entire new file again because the new file would be offset by the new data and a new hash would happen.
The best solution is delta versioning solution to de-duplicate this file.
It has two different methods:
1) Block-level delta vesioning
2) Sub-block-level delta versioning.

Block-level delta versioning:
This method is like snapshot that I explained before. This method monitors all updates on disk at block level and stores only the data that changed in relation to original data. This method really decreases the amount of data to send to Disaster Recovery site and it's really efficient.

Sub-block-level delta versioning (Microscan):
It's the same as the Block-level delta versioning but it's more efficient and it works in byte level and not block level. In figure, if we use our previous example, you add one character to the original file which means you need a single 512 bytes (one sector) to store the changes to the original file. Now, if you use an array replication software to send the file to Disaster Recovery site, it will send an entire 32K disk track because that's the minimum level of block definition for array-based solution. If we use a file level monitoring solution, we still need to send the entire 8K because the file system needs 8K to write the data.

Microscan monitors changes at a disk sector level (1 sector=512 bytes) and replicate only the 512-bytes changed delta. A block of data in Windows, for example, is 4KB (8 x 512). So, in Block-level delta versioning method, when a 512 bytes inside a 4KB block is changed, it will send entire 4KB block. But in Microscan or Sub-block-level method, only a 512 bytes changed data will replicate. Therefore, you save a lot of white space. Microscan is really the best efficient method when you are using monitoring solution in the data being updated.   
Hope you enjoyed.
Regards,
Khosro Taraghi

What Is Data De-Duplication and How It Works?

Hello all, 

Data De-Duplication means comparing objects(files and blocks) and removing all duplicated objects and  keep only unique objects. So, the results is a smaller group of data(files and blocks). For example,

 

If you look at the picture above, de-duplication has removed all duplicated data. The result is a smaller group of data. And, of course, there is a way to reproduce the original data. I’ll explain that.

What is the advantage of De-duplication?

1.  Reduced backup cost:  you can save a huge amount of data in terms of size when you save de-duplicated data to tape or sending backups to remote site using WAN or LAN.
2. Reduced WAN and LAN Bandwidth: Using de-duplicated data, you can save bandwidth and reduce the cost of using WAN if you send your data to a remote site. 
3. Reduced hardware cost: you need less tape and harddisk  
4. Increased efficiency of storage  

De-Duplicated methods:

1. File-based comparison and then compression:This method, which is an old method, uses operation system or application to compare files, for example, comparing the name, size, type, and date of modification. If all parameter matches, you can remove one of them. If you use a file-compression method, you can save more space too. In this method, the de-duplication ratio would be 2:1 or 3:1, which 50 % less data. This can be done through a script or operation system and it’s free.  
2. File level hashing:It’s like file-based method but it’s more intelligent. File level hashing creates a unique mathematical hash for files. Then, it will compare the hashes for new files with the originals one. If the hashes match each other, it means the files are the same and it can be removed. This method requires an index table to store the hashes and it can be referenced quickly for match. Usually, the indexes are stored in RAM and they are very quick and don’t slow down the process of hash look up. 
3. Block level hashing:  It’s the same concept as File level hashing but it works with the block of data. So, it’s independent of file system in OS or files themselves. So, the block of data means the way that data is stored on disk and it doesn’t care about the type of data. De-Duplication uses hash for blocks of data and compares every new block of data being stored through the de-duplication and it will remove the equal blocks.  
4. Sub-Block level hashing: It works like Block level hashing but it’s much more sufficient. This is the most common method that used in enterprise today. It divides or slices a block of data into a set of sub blocks with a specific size. For example, it gets a 64KB block of data and creates 4 segments of 16KB each. Then it creates a unique hash for each slice or chunk of data.

The hashes are stored in index hash table and it starts to compare the hashes. As you see in picture, the hashes for chunk number 1 and 3 are equal. So, it removes the duplicated chunk of data for segment 3 and it puts its hash as a pointer so the original data can be restored later.
 Please look at the picture below:

Segment 3 is removed and replaced with the hash as a pointer and now the original file takes less space. The original file can be rebuilt by replacing the hash for segment 3 with the data in segment 1.

       5.   Delta versioning: I will explain this method in my next blog post since it needs more explanations.

Hope you enjoyed.

Regards,
Khosro Taraghi