In VariableHashEngine.java (line 61) :
public List<Finger> getChunks(byte [] b,String lookupFilter,String uuid) throws IOException { final ArrayList<Finger> al = new ArrayList<Finger>(); ff.getChunkFingerprints(b, new EnhancedChunkVisitor() { public void visit(long fingerprint, long chunkStart, long chunkEnd, byte[] chunk) { byte[] hash = getHash(chunk); Finger f = new Finger(lookupFilter,uuid); f.chunk = chunk; f.hash = hash; f.len = (int) (chunkEnd - chunkStart); f.start = (int) chunkStart; al.add(f); } }); return al; } After Chunking Chunks' Order Output In File DeduplicationSDFS implements the deduplication process within the file in SparseDedupFile.java. In the public void writeCache(WritableCacheBuffer writeBuffer) function, the dedup in the file is implemented as follows:
HashMap<ByteArrayWrapper, Finger> mp = new HashMap<ByteArrayWrapper, Finger>(); for (Finger f : fs) { ByteArrayWrapper ba = new ByteArrayWrapper(f.hash); Finger _f = mp.get(ba); if (_f == null) { f.claims = 1; mp.put(ba, f); } else { _f.claims++; mp.put(ba, _f); } }ByteArryWrapper is a derivative of Byte Arry. It does not have the ability to deduplicate chunks. In the content, ba is the serialization result of the hash of the currently processed chunk. mp.get() will find out whether the hash (key) exists from the HashMap. The hash (key) and finger (value) are inserted into the HashMap when they are not present. The declaration variable of the finger records the number of times the finger appears in the current HashMap.
Output PositionIn SparseDedupFile.java, add bit string to hex string function bytesToHex after line 399:
private final static char[] hexArray = "0123456789ABCDEF".toCharArray(); public static String bytesToHex(byte[] bytes) { char[] hexChars = new char[bytes.length * 2]; for ( int j = 0; j < bytes.length; j++ ) { int v = bytes[j] & 0xFF; hexChars[j * 2] = hexArray[v >>> 4]; hexChars[j * 2 + 1] = hexArray[v & 0x0F]; } return new String(hexChars); }Add output function after line 414:
String metaDataPath = "/sdfsTemp/metaData/" + this.GUID; for (Finger f : fs) { try { FileWriter fw = new FileWriter(metaDataPath, true); fw.write(Integer.toString(f.start)); fw.write("\t"); fw.write(Integer.toString(f.len)); fw.write("\t"); fw.write(Integer.toString(f.chunk.length)); fw.write("\t"); fw.write(bytesToHex(f.hash)); fw.write("\n"); fw.close(); } catch (IOException e) { e.printStackTrace(); } ···It will output the after chunking chunks' order in /sdfsTemp/metaData/GUID for any single sile.
Before Deduplication Chunks' Order Output (HCServiceProxy Progress) Between Files DeduplicationIn SparseDedupFile.java's writeCache function, after deduplicate progress in single file by HashMap. The Chunks in HashMap (only contains unique chunks) will be added to a ThreadPool to do the next step deduplication (DSE between files deduplication) by FingerPersister runnable function which was designed in class Finger. In this function, every single chunk will be added to HCServiceProxy to find dup and write unique chunk to chunk store. Because this function will be done by multi-thread function, and all that work needs to call HCServiceProxy.writChunk function. So we will add output whenever that function was called.
Output PositionIn HCServiceProxy.java, add bit string to hex string function bytesToHex after line 49:
private final static char[] hexArray = "0123456789ABCDEF".toCharArray(); public static String bytesToHex(byte[] bytes) { char[] hexChars = new char[bytes.length * 2]; for ( int j = 0; j < bytes.length; j++ ) { int v = bytes[j] & 0xFF; hexChars[j * 2] = hexArray[v >>> 4]; hexChars[j * 2 + 1] = hexArray[v & 0x0F]; } return new String(hexChars); }After line 245, add two kinds of output for HCServiceProxy :
All chunks (at /sdfsTemp/dedup/ComeChunks-HC-All-Chunks)
The chunks in a single file (at /sdfsTemp/dedup/ComeChunks-HC-uuid).
The chunk store on the server side is based on the implementation of the AbstractChunkStore template class as follows: