Entropy Indicates File Boundaries
One of the problems in file carving is to properly identify the borders of the former files. This especially holds true in case of fragmentation. In their submission to the DFRWS 2006 challenge Klayton Monroe and Jay Smith of KoreLogic Security and Andy Bair of MITRE (at that time) calculate the block-wise entropy to detect file boundaries and identify blocks which are unlikely to belong to a carved file.
The team used the free FTimes utility and the power of its XMagic language to survey the topology of the raw data stream. XMagic is also capable of calculating frequencies of various C-style data types along with a block-wise average and entropy.
Their presentation contains some interesting examples demonstrating how "defects" in carved files are indicated by sudden changes in entropy (slides 34 et. seqq.)
