The answer is that this technology is so deeply embedded within Windows and Windows applications that you might not be able to avoid it. For example, Office documents are stored in OLE compound file format and, despite the fact that the latest version of Office has a new “open” XML file format, compound documents are going to be a nuisance for some time to come. HTML help is also stored using structured storage – so if you want to read .CHM files you need to master structured storage. Another example is the Thumbs.db file that Windows generates to store all of the thumbnail images used to display graphics files in a directory. It might well end in “db”, but it isn’t a standard database file – it’s structured storage.
There are no structured storage classes within the .NET framework, so to work with it you’ll have to implement your own, and this means dealing with several COM Interfaces. Initially this looks very difficult but, with some help, it begins to seem more manageable.
PInvoke the easy way
Accessing the information in Thumbs.db provides an excellent example of working with structured storage. In most cases you are going to want to read and extract the information within a structured storage file, and reading the Thumbs.db file demonstrates just how to do this. It is also an excellent example of how to work with COM interfaces in C# in general.As well as Interfaces, there are also a lot of “helper” functions which can be called using PInvoke. For example, before you start processing a file as if it was structured storage you should use the StgIsStorageFile function to test that it really is. The definition of this function is simply:
[DllImport(“ole32.dll”)] static extern int StgIsStorageFile( [MarshalAs(UnmanagedType.LPWStr)] string pwcsName);To use it to test if a file is structured storage you would use:
string file = @”path\Thumbs.db”; int result = StgIsStorageFile(file);The result is 0 if the file is a structured storage file, 1 if it isn’t and 0x80030006L if the file isn’t found. It should return a 0 if you specify a Thumbs.db file that exists. Remember that Thumbs.db files are system files and therefore hidden unless you use the Tools,Folder Options command to make them visible. You also have to deselect “Do not cache thumbnails” option to make sure that Windows generates the Thumbs.db file in that directory.
This is easy enough, but it’s even easier if you know about PINVOKE.NET – a Wiki dedicated to providing a complete library of Dllimport, interface, constant and struct definitions needed to work with the Windows API from a managed environment. All you have to do is enter the name of the API function, interface, constant or struct you are interested in, and you will most often be presented with a ready-made definition. Sometimes the definitions are clearly well used and tested as evidenced by the availability of examples and user comments. Other, less well-trodden paths are also evident with just brief definitions and a plea to add some sample code and notes. In these areas you do have to read and check that the definitions make sense – surprisingly they usually do! You might also want to change the way some are defined to fit in with how you want to use them. In short this isn’t a complete solution but it saves a huge amount of time, and if you do check that something works please do contribute to the Wiki. It also raises the question of why Microsoft hasn’t done as much to make PInvoke easier to use. In the rest of this article the definitions available at PINVOKE.NET are used and only changes or specific uses of parts of the definitions will be detailed.
Open the file
In most cases the first thing that you need to do when working with a structured storage file is to open it. Another helper function does this job – StgOpenStorage. Its definition can be found at PINVOKE.NET:[DllImport(“ole32.dll”)] static extern int StgOpenStorage( [MarshalAs(UnmanagedType.LPWStr)] string pwcsName, IStorage pstgPriority], STGM grfMode, IntPtr snbExclude, uint reserved, out IStorage ppstgOpen);At this point we hit our first “definition cascade”. The function is simple enough, but it makes use of the new types STGM and IStorage. STGM is just a set of constants that control the SToraGe Mode, i.e. the access mode. You could look these constants up in the header file and write them explicitly into the program, but PINVOKE.NET has a very nice definition for STGM, which starts:
[Flags]
public enum STGM : int
{
DIRECT = 0x00000000,
TRANSACTED = 0x00010000,
...
The [Flags] attribute means that the members can be treated as bit fields and combined together using logical operations to set multiple conditions.
The STGM enumeration would have been easy to create manually, but not so the IStorage interface. Thankfully there is a translation in both C# and VB at PINVOKE.NET which starts:
[ComImport]
[Guid(“0000000b-0000-0000-C000-000000000046”)]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
interface IStorage
{
void CreateStream(
/* [string][in] */ string pwcsName,
/* [in] */ uint grfMode,
/* [in] */ uint reserved1,
/* [in] */ uint reserved2,
/* [out] */ out IStream ppstm);
You can copy and paste this and the STGM enumeration into your program. It is clear that the IStorage definition at PINVOKE.NET isn’t much used, so you need to check it as you use each function. Reading through the definition, IStorage also seems to introduce new types that should need explicit definitions. The first is FILETIME used in SetElementTimes:
void SetElementTimes( /* [string][unique][in] */ string pwcsName, /* [unique][in] */ FILETIME pctime, /* [unique][in] */ FILETIME patime, /* [unique][in] */ FILETIME pmtime);We don’t need to add a definition for this because it’s already defined within ComType:
void SetElementTimes( /* [string][unique][in] */ string pwcsName, /* [unique][in] */ System.Runtime.InteropServices.ComTypes.FILETIME pctime, /* [unique][in] */ System.Runtime.InteropServices.ComTypes.FILETIME patime, /* [unique][in] */ System.Runtime.InteropServices.ComTypes.FILETIME pmtime);Similarly the Stat function introduces the STATSTG struct:
void Stat( /* [out] */ out STATSTG pstatstg, /* [in] */ uint grfStatFlag);As this too is defined in ComTypes, we simply change this to:
void Stat( /* [out] */ out System.Runtime.InteropServices. ComTypes.STATSTG pstatstg, /* [in] */ uint grfStatFlag);Fully qualified names are necessary because FILETIME and STATSTG are ambiguous. The new interface IStream, which is introduced in a number of methods, is also defined in ComType, but as it is unambiguous we can deal with it by adding:
using System.Runtime.InteropServices.ComTypes;Finally we have the IEnumSTATSTG interface introduced in the all-important EnumElements function:
void EnumElements( /* [in] */ uint reserved1, /* [size_is][unique][in] */ IntPtr reserved2, /* [in] */ uint reserved3, /* [out] */ out IEnumSTATSTG ppenum);This isn’t available in ComTypes or anywhere else, so we have to make use of PINVOKE.NET’s definition, which starts:
[ComImport] [Guid(“0000000d-0000-0000-C000- 000000000046”)] [InterfaceType( ComInterfaceType. InterfaceIsIUnknown)] public interface IEnumSTATSTGIf you copy and paste this into the program you will discover that it too defines types which aren’t yet defined. At this point in the development of this sort of project you begin to think that you will never have everything defined, as each time you introduce something you need more definitions. Of course it does end, and in this case all that is needed is to change the Next method to read:
[PreserveSig] uint Next( uint celt, [MarshalAs(UnmanagedType.LPArray), Out] System.Runtime.InteropServices. ComTypes.STATSTG[] rgelt, out uint pceltFetched );This makes use of the STATSTG structure defined in ComTypes. At this point everything is defined, and we can move on to start making use of the new interfaces and types. The definitions may need some fine-tuning, but the effort PINVOKE.NET has saved just in typing is evident.
At long last we can now open a structured storage file:
IStorage Is; result = StgOpenStorage(file, null, STGM.READWRITE | STGM.SHARE_EXCLUSIVE, IntPtr.Zero, 0, out Is);It is assumed that “file” contains the name of a Thumbs.db file complete with path. Notice that the file is opened in ReadWrite and Share_Exclusive mode. This is usually what you need to do, but the whole subject of modes of opening structured storage is complicated, and if you get an error message when you try to open a file this is usually the source of the problem.
Using IStorage
The StgOpenStorage function returns an IStorage interface – the good news is that this works and we can now start to manipulate the structure storage using the functions it defines. In most cases the first thing we need to do after opening a structured storage file is to find out what’s stored in it. This is a two-step process – first get an enumerator and second use the enumerator’s Next function to retrieve STATSTG structs which describe each storage element. Getting the enumerator is easy:IEnumSTATSTG SSenum; Is.EnumElements(0, IntPtr.Zero, 0, out SSenum);We can now use a loop to step through the STATSTG structs. The only minor complication is that we have to pass an array of STATSTG structs and say how many we want to retrieve at each use of Next. In most cases it’s simpler to retrieve just one:
System.Runtime.InteropServices. ComTypes.STATSTG[] SSstruct = new System.Runtime.InteropServices. ComTypes.STATSTG[1];The Next function tries to return the number requested (one in this case), but will return fewer if there aren’t enough remaining in the enumeration. The number actually returned is given by the final parameter and this is used to stop the iteration:
uint NumReturned;
do
{
SSenum.Next(1, SSstruct,
out NumReturned);
if (NumReturned != 0)
{
textBox1.Text = textBox1.Text +
SSstruct[0].pwcsName + “ “ +
SSstruct[0].type.ToString()
+ Environment.NewLine;
}
}while(NumReturned>0);
In this example the name of each of the structured storage elements is added to a Textbox along with the element’s type. If you run this on a Thumbs.db file you will see that there are elements called 1,2,3 and so on, and one called Catalog. All of these are of type 2. To discover what the type codes mean we need the STGTY enumeration. Unfortunately, at the time of writing, PINVOKE.NET didn’t have a suitable definition, but one is easy to construct from the C++ definition given in the documentation:
enum STGTY :int
{
STGTY_STORAGE = 1,
STGTY_STREAM = 2,
STGTY_LOCKBYTES = 3,
STGTY_PROPERTY = 4
};
You can see that there are four possible types stored in a structured storage file. All of the elements in Thumbs.db are streams, i.e. data files that can be written and read in the usual way. Elements of type 1 are storage objects, and these can contain complete filing systems just like the root storage object we have opened. You can think of this as being the analogy of sub-directories within a root directory. What this means is that to find out everything stored in a structured storage you have to open each storage object in the top-level storage object, and enumerate what they contain. Clearly this is a job for a recursive function, but given that Thumb.db files only contain a single top-level storage object, this is an exercise left for the reader. The remaining two possibilities are special types of “file”. Type 3 is a byte array and type 4 is a property set of property name value pairs. In most cases you can ignore these, but if you need to know more look up the interfaces and helper functions with PROP included in their name and ILockBytes.
Reading a stream
The Thumbs.db file always contains a Catalog stream and this contains information about the thumbnails defined within the structured storage. The format of Catalog isn’t completely determined, but it starts with a header corresponding to the struct:public struct CatalogHeader
{
public short num1;
public short num2;
public int thumbCount;
public int thumbWidth;
public int thumbHeight;
}
First we need to open the stream using the IStorage function OpenStream:
IStream CatStream; Is.OpenStream(“Catalog”, IntPtr.Zero, (uint)(STGM.READWRITE | STGM.SHARE_EXCLUSIVE), 0, out CatStream);This returns an IStream interface, which is used to read the raw bytes stored in the stream. At this point we have the problem of reading in a struct stored as binary data. There are a number of ways of doing this, but no single “best”. If you have only numerical data in a struct then a common method is to use a BinaryReader. First we need a byte buffer and a variable to hold the number of bytes actually read:
byte[] buf = new byte[1000]; uint count;Next we create an instance of the structure and read in all of the bytes that correspond to it:
CatalogHeader CH = new CatalogHeader(); CatStream.Read( buf, Marshal.SizeOf(CH), new IntPtr(&count));Notice the slightly strange way the final parameter is passed. If you look up the definition in the documentation it says:
pcbRead
A pointer to a ULONG variable that receives the actual number of bytes read from the stream object.
…which suggests that you pass a pointer type. However, the Intellisense prompting reveals that the function is actually expecting an IntPtr. There is most definitely a better way of defining this Interface function, e.g. marshal the parameter as (out) ulong, but as it’s predefined in ComType it’s simpler to just use it “as is”.
The slightly unnerving:
new IntPtr(&count));…does the job of passing a “pointer” so that the function can change the value of count. Now that we have a buffer with all of the data relevant to the header we can create a BinaryReader based on it, and proceed to read the various data types from it:
BinaryReader head = new BinaryReader( new MemoryStream(buf)); CH.num1=head.ReadInt16(); CH.num2=head.ReadInt16(); CH.thumbCount=head.ReadInt32(); CH.thumbWidth=head.ReadInt32(); CH.thumbHeight=head.ReadInt32(); head.Close();This method works reasonably well as long as the structure has a fixed size and is composed of types that the BinaryReader supports. When it comes to reading the rest of the Catalog this isn’t the case. After the header there are “thumbCount” records, one for each thumbnail corresponding to the struct:
public struct CatalogItem
{
public int num1;
public int itemID;
public DateTime Modified;
public string filename;
public short num2;
}
The DateTime field is stored as a 64-bit FileTime format value and the filename field is stored as a C-style, variable length, null-terminated string. To read this struct we need a different approach, and what could be simpler than using a BitConvertor:
CatalogItem CI = new CatalogItem(); CatStream.Read(buf, 16, new IntPtr(&count)); CI.num1 = BitConverter.ToInt32(buf, 0); CI.itemID = BitConverter.ToInt32(buf, 4); CI.Modified = DateTime.FromFileTime( BitConverter.ToInt64(buf, 8));The idea is to read in the first 16 bytes, i.e. up to the start of the string, and then simply convert each group into the appropriate type. The DateTime object is constructed directly from the 64 bits that make up the FileTime value using the very useful FromFileTime static method.
Reading in the string can’t be done easily in one go because we don’t know how many bytes it corresponds to. We are forced to write an explicit loop and test for the null bytes at the end of the string:
StringBuilder temp =
new StringBuilder(50);
do
{
CatStream.Read(buf, 2,
new IntPtr(&count));
temp.Append(
BitConverter.ToChar(buf, 0));
} while (buf[0] != 0 || buf[1]!=0);
Notice that in this case a character corresponds to two bytes because this is a Unicode string, and indeed ToChar converts a two-byte value to a Unicode character. Now we can store the string in the struct and read the final Int16:
temp.Length = temp.Length – 1; CI.filename = temp.ToString(); CatStream.Read(buf, 2, new IntPtr(&count)); CI.num2 = BitConverter.ToInt16(buf, 0);Notice the need to reduce the size of the string by one to remove the null from the end.
This reads just a single record from the Catalog, but converting it into a function and iterating through all of the records isn’t difficult.
Reading the JPEG
By comparison with reading the Catalog, reading one of the thumbnail files is very easy. If the files being thumbnailed are JPEGs, then the thumbnails are also JPEGs, but with 12 bytes of additional data added to the start.So to read the thumbnail corresponding to the element called “1” and store the result in a .JPG file, all you have to do is open a new IStream:
IStream ThumbStream; Is.OpenStream(“1”, IntPtr.Zero, (uint)(STGM.READWRITE | STGM.SHARE_EXCLUSIVE), 0, out ThumbStream);Read the first 12 bytes and throw them away:
byte[] bits = new byte[1000]; ThumbStream.Read(bits, 12, new IntPtr(&count));Open a BinaryWriter for the new JPEG file:
BinaryWriter JPEGfile = new BinaryWriter(File.Open(“1.jpg”, FileMode.Create));Finally keep reading blocks of bits and writing them out to disk until there are no more to read:
do
{
ThumbStream.Read(bits, 1000,
new IntPtr(&count));
JPEGfile.Write(bits, 0, (int)count);
} while (count > 0);
JPEGfile.Close();
You now have a standard JPEG file containing the thumbnail stored on disk.
Creating a BitMap
If you want to work with the thumbnail in memory you probably want to convert it into a BitMap. This is relatively easy, but first we need to reposition the stream back to the start, or in this case to the 12th byte.This can be done using the IStream Seek function, which makes use of the Stream_Seek enumeration to indicate where the location is relative to. There isn’t a predefined enumeration for this, but it’s easy to add one:
enum STREAM_SEEK : int
{
STREAM_SEEK_SET = 0,
STREAM_SEEK_CUR = 1,
STREAM_SEEK_END = 2
};
The call to the Seek function to position the reading location at the 12th byte is:
uint position; ThumbStream.Seek(12,(int) STREAM_SEEK.STREAM_SEEK_SET, new IntPtr(&position));Notice that we have to cast the Stream_Seek type to int – making it almost not worth using the enumeration. We now need to read in the entire JPEG file into a byte array and to do this we need to know the size of the stream. This can be found using the Stat function:
System.Runtime.InteropServices.ComTypes.STATSTG fileinfo; ThumbStream.Stat(out fileinfo, 0);The StatSTG structure, fileinfo, gives lots of information about the stream including cbSize, which is the stream size in bytes. We can now create a buffer of exactly the right size and read the data in:
byte[] BitmapData = new byte[fileinfo.cbSize-12]; ThumbStream.Read(BitmapData, BitmapData.Length, new IntPtr(&count));To create a BitMap from this data we first need to convert it into a Stream, a MemoryStream to be precise, and then use the appropriate constructor:
MemoryStream BitmapStream = new MemoryStream(BitmapData); Bitmap Jmap = new Bitmap(BitmapStream); pictureBox1.Image = Jmap; }To display the result the Bitmap has been assigned to a PictureBox’s Image property. Similar techniques can be used to save an existing JPEG back into the stream to change the thumbnail.
Conclusion
There are many other structured storage tasks, such as creating, writing and even deleting IStream elements, but the basic ideas are the same as for reading IStream elements.In principle when you investigate a structured storage file the IStreams that you find stored inside should be in standard format. That is, if you find a Tiff stored within a Word document then you can open and read it as if it was a standard Tiff stored in a file all on its own.
In practice slight modifications to the stored files often make this more difficult, and there is very little official documentation on how the formats have been modified. Working with structured storage isn’t difficult in principle, but in practice it can be frustrating.
Harry Fairhead is the author of various books and articles on computer architectures. His current interests include systems and embedded programming.
Comments