Reading, writing and photo metadata

Metadata means “data about data”, and while it sounds like a boring theoretical idea, it is the key to new and powerful ways of organising information. Metadata is behind the next revolution on the web – the so called “semantic web” – and it’s at the core of Windows Vista and Windows 7 in the form of the Vista Property System and Windows Desktop Search. The idea is a very simple one, and we have all been using metadata for a long time – the size of a file, the file name, and the number of words in a document are all examples of metadata. Put simply, metadata is just information about what is stored in a file. Other names for metadata include properties and tags. The idea is that assigning properties and tags to files makes it much easier to find and organise them using a search engine – like the one built into Vista and Windows 7.

One of the most useful examples of metadata in action is the copious information that most digital cameras store along with each picture. This includes the make of the camera, the exposure, the photographer, the date and, in the case of GPS equipped cameras, the exact location (in latitude/longitude co-ordinates) that the photo was taken. Clearly if you are working on any application that involves digital photography, you need to know how to handle metadata. In principle we could tackle the problem of reading and writing photo metadata by implementing general metadata reading and writing methods, but for the moment there isn’t a managed way of working with the new Vista Property system. This can be retrofitted to Windows XP, and in principle it provides a unified way of working with metadata of all types, but Microsoft has opted to not only implement it as a COM interface, but has also neglected to provide any managed access. This is disappointing to say the least – given how many developers have backed .NET, you would think that Microsoft would do the same.

However, it is worth knowing how the broader metadata system works in principle, because this is the way of the future – see “The Story of Metadata”.

NET 2.0 properties

There is support for access metadata in bitmap images in .NET 2.0 and above. Although primitive and lacking many facilities, it’s simple and it might be all we need. Start a new Windows project and place a button on the form. Add the following using:

using System.Drawing.Imaging;

In the button’s click event handler add:

Bitmap MyPhoto =
    new Bitmap(@"myphoto.JPG");
PropertyItem[] props =
    MyPhoto.PropertyItems;
foreach (PropertyItem prop in props)
{
    MessageBox.Show(
    	prop.Id.ToString());
}

Replace myphoto.JPG by a complete path to a JPEG image. The PropertyItem array contains all of the properties that the photo supports. A PropertyItem has just four members:

• Id – gets/sets the ID
• Value – the value of the property
• Type – the type of the Value
• Len – the length of the Value

The example code simply lists all of the property IDs. This is very simple but not really very useful. To get any further you need to know the ID codes for each of the properties. Many of these are listed in the informal specification which can be downloaded from the unofficial EXIF web site, www.exif.org. Notice that manufacturers add custom codes which are not listed here. There is also a good list on the MSDN website. For example, ID 272 is the camera model type, and we can retrieve just this property using:

const int IDCamModel=272;
PropertyItem Make =
    MyPhoto.GetPropertyItem(
    	IDCamModel);

You can examine some of the metadata stored along with a photo by using the usual Properties dialog box

Now that we have the camera model property in the PropertyItem, the next question is how to get the value. The problem here is that there is no attempt made to process the data into a usable form. The PropertyItem.Value is always returned as an array of bytes, and it’s up to you to use the Type and Len properties to process it into a more useable form. For example, the camera model property has a Type value of 2, which according to the documentation is a null terminated ASCII encoded string – i.e. a C style string. Converting a zero or null terminated ASCII byte array into a string is very easy using the Encoding class:

Encoding ascii=Encoding.ASCII;
string CamModel=ascii.GetString(
    Make.Value,0,Make.Len-1);
MessageBox.Show(CamModel)

Notice that we have to exclude the final byte from the operation as it contains the zero which marks the end of the string. Although converting byte arrays to strings and vice versa is a very common operation, it is surprising how many programmers miss the fact that the Encoding class exists – it is the right way to do the job.

It isn’t difficult to see how to handle all of the properties returned in a PropertyItem object. The project involves setting up a file of property ID constants, writing methods to convert the Value byte array into the .NET data type as indicated by the Type property, and then wrapping the whole lot into an easy to use class. Not difficult, but not particularly interesting either, and at the end of the day you still have some problems. In particular, for reasons that are difficult to understand, the implementation of PropertyItem doesn’t have a public constructor. This means that you can’t write:

PropertyItem MyItem =
    new PropertyItem();

And this in turn means it’s difficult to add your own properties to a photo. There is a way round this problem. You can get a PropertyItem from the bitmap and then change its ID, Value, Type and Len properties to be the new property that you are trying to save to the file. This works, but to write the property to the photo you have to write the entire file out, and this means recoding the photo with the consequent loss of quality that lossy JPEG compression implies. It’s also worth adding that trying to write properties using this method has varied success depending on the exact file format and the property being written. In short, the .NET 2.0 approach works reasonably well if you want to read a few properties but it is not a good way of working if you want to write properties.

You can display a Bitmapsource in a Bitmap

.NET 3.0 properties

When .NET 3.0 was released, it brought a much better approach to properties. Start a new C# Windows application and add a reference to the following new .NET 3.0 Framework assemblies: PresentationCore, PresentationFramework and WindowsBase. If you can’t find them then you haven’t downloaded and installed .NET 3.0. You also need to add:

using System.IO;
using System.Windows.Media;
using System.Windows.Media.Imaging;

All of the new .NET 3.0 imaging classes are wrappers for the new Windows Imaging Component (WIC), which is installed as part of .NET 3.0 and is also available as a separate download – it’s pre-installed under Vista. The key idea is that each image type has an associated Decoder class that knows how to read the image and any metadata associated with the image. For example, The JpegBitmapDecoder knows all about how to read a JPEG file via a data stream:

Stream imageStream = new FileStream(
    @"myphoto.JPG",
    FileMode.Open,
    FileAccess.Read,
    FileShare.Read);
JpegBitmapDecoder decoder = new
    JpegBitmapDecoder(
    	imageStream,
    	BitmapCreateOptions.
    		PreservePixelFormat,
    	BitmapCacheOption.Default);

Following this, provided the specified file exists, the Bitmap and all its associated data are stored in the decoder. The actual bitmap data is in the Frames array property. It’s an array because some bitmap formats, e.g. GIF, allow multiple bitmaps per file to create animations and so on. In the case of JPEGs only Frame[0] should have any image data stored in it. Other important properties are Metadata, i.e. where all the metadata lives, and Preview and Thumbnail. At this point you might be tempted to try to display the image stored in Frame[0], but this turns out to be difficult because it isn’t a GDI/GDI+ bitmap – it’s a DirectX bitmap. If you are working with a WPF project then you can simply use an Image control to view the photo. If you are working with .NET 2.0 forms then you have to transfer the image data from to a .NET 2.0 Bitmap object and this isn’t trivial. The following function does some of the work:

private Bitmap BSourceToBitmap(
    BitmapSource BS)
{

First we need to compute the “stride” of the image, i.e. how many bytes of storage correspond to a single row of pixels:

int stride=BS.PixelWidth*
    BS.Format.BitsPerPixel/8;

The stride has to be rounded up to the nearest 4-byte boundary:

stride=stride+(stride%4)*4;

Next we need a byte array to hold the pixel data and then copy all of the raw pixel data into it:

byte[] Pixels = new byte[
    stride*BS.PixelHeight*
    	BS.Format.BitsPerPixel/8];
BS.CopyPixels(Pixels, stride, 0);

Now we create a Bitmap object with the same size, format and using the raw data. This is an “unsafe” operation and has to be in an unsafe block:

unsafe
{
    fixed (byte* pPixels = Pixels)
    {
    	IntPtr ptr =
    		new IntPtr(pPixels);
    	Bitmap bitmap = new Bitmap(
    		BS.PixelWidth,
    		BS.PixelHeight,
    		stride,
    		System.Drawing.Imaging.
    		PixelFormat.Format24bppRgb,
    		ptr);
    	return bitmap;
    }
}

This function works as long as the photo is using 24-bit colour. The problem with creating a completely general function is that you have to specify the pixel format for the new Bitmap object, and the new .NET 3.0 PixelFormat isn’t compatible with the existing .NET 2.0 PixelFormat enumeration. To make this function completely general you would need to write a conversion routine – possible but tedious. An easy, but possibly inefficient, alternative is to simply load the image again into a Bitmap object. With a Bitmap object we can now display the photo in a PictureBox:

pictureBox1.Image =
    BSourceToBitmap(
    decoder.Frames[0]);

Reading Metadata

Returning to our original quest for metadata, the new BitmapSource object makes this much easier. It has a Metadata property that returns an ImageMetadata object. This in turn has a set of properties which give values for most of the common tags. For example, to get the camera model all you have to do is use the CameraModel property:

BitmapMetadata Mdata =
    (BitmapMetadata)bitmapSource.
    Metadata;
MessageBox.Show(
    Mdata.CameraModel.ToString());

Notice that we need the cast to BitmapMetadata because each image type has its own implementation of ImageMetatdata with its own set of properties. In theory you can set properties simply by assigning to them. A BitmapMetadata object has nearly all of the properties you need – Author, Copyright, Title and so on, but there are lots of properties that are not included as standard, and there is a fairly easy way of getting at these. The GetQuery and SetQuery methods provide completely generally ways of getting and setting a property. The only real difficulty in using them is discovering what properties are supported and what they are called. A partial answer to the naming problem can be found by searching the Microsoft website for “Photo Metadata Policy”. You can always use EXIF Id codes to retrieve a property. For example to retrieve Id 272 (i.e. the camera model) you would use:

object t = Mdata.GetQuery(
    @"/app1/ifd/{ushort=272} ");
MessageBox.Show(t as string);

A similar format for all of the query strings allows you to retrieve any EXIF as documented in the Metadata Policy document. The metadata is structured in a way that is similar to an XML document, and /app1/ifd is where the EXIF metadata starts. Different tags provide access to different metadata stores, and this is poorly documented. There is more than one way to store metadata in a photo, and it is possible to encounter multiple formats within a single file. For example, if you use Vista to annotate a photo then the metadata will be stored in an XMP block and, for example, the camera model can then be retrieved using:

object t = Mdata.GetQuery(
    @"/xmp/tiff:model");

Vista, in its attempt to unify all metadata, even goes to the trouble of converting all of the EXIF tags it knows into XMP. So if a photo has been edited in Vista you can expect to see EXIF and XMP tags.

Of course in a general situation you don’t know in any photo how the metadata is actually stored and hence what query string will return the metadata you want. The solution is to use “conflict resolution”. This is how Vista hopes to manage the metadata mess. For example, if you use the query:

object t = Mdata.GetQuery(
    "System.Photo.CameraModel");

…then the system will first look in the XMP metadata, if any, to see if there is a model property, and if not it looks in the EXIF metadata. Writing a property using system properties also attempts to keep all of the metadata stores synchronised and valid. Clearly, if at all possible, you should always use system properties as it provides a unified approach to metadata. However, we are still in a state of transition and not all of the properties that EXIF supports are actually “resolved” to a system property. For example, at the time of writing the system property for the F number used to take the photo:

object t = Mdata.GetQuery(
    "System.Photo.FNumber");

…fails with an exception. However:

object t = Mdata.GetQuery(
    @"/app1/ifd/exif/{ushort=33437}");

…which is the EXIF property that FNumber is supposed to check returns with a numeric value – one that seems to have no relationship to the FNumber used – but a result nonetheless.

Microsoft’s own metadata editor ImageInfo seems to be able to do more with metadata than .NET allows

Writing metadata

You can, in theory, use SetQuery or any of the standard properties to change the value of metadata – but notice that some values are treated as read-only by the system and hence aren’t write-able in principle. There is also the problem that the entire BitmapMetadata object is set to frozen, and this stops any modification to the metadata. You can create a write-able version of the metadata using the Clone method, and then you can change some properties directly and some using SetQuery as appropriate. For example, to set the author metadata you can use:

BitmapMetadata tempMeta =
    Mdata.Clone();
string[] values ={ "mike james" };
tempMeta.Author=new
    System.Collections.ObjectModel.
    ReadOnlyCollection<string>(values);

…but if you now try to set this metadata to a Metadata property you discover that this is read only. In fact no matter how you wriggle you can’t seem change the metadata in this way. Even if you could, you have the problem that after changing the metadata the entire file has to be recoded and stored in a new file. Not good given that every JPEG encoding results in a loss of quality.

As an alternative we can try an in-place writer which sounds very promising:

InPlaceBitmapMetadataWriter IPW =
    decoder.Frames[0].
CreateInPlaceBitmapMetadataWriter();
IPW.Comment = "a nice picture";
IPW.TrySave();

However, this either throws an exception or simply fails to store the changes back to the original file (which has to be open as read/write). There seems to be evidence that the in-place writer only works if there is space in the file to store the data, i.e. you can’t increase the length of a string using it. I can’t confirm this because despite my best efforts I have failed to store any metadata changes using this method.

Where next?

The photo metadata system supplied with .NET 3.0 seems to be a complete mess. The documentation is the most misleading I have ever encountered, and it’s difficult not to draw the conclusion that it is trying to cover up the fact that this subsystem simply doesn’t work as advertised. What is surprising is that there is clearly more on offer if you are prepared to move to unmanaged code and the Vista Shell, but this isn’t what we signed up for when we adopted .NET.

There are some excuses for legacy systems still not having managed interfaces, but why is Microsoft still introducing essentially COM-based APIs with hardly any managed code support? This is worse than useless, and given Microsoft’s official line to support managed code in Vista and beyond, it’s pretty unforgivable.

Appendix: The story of metadata

The big problem with metadata is where do you store it? If you opt to store it within the file, then adding a custom field changes the format of the file and perhaps makes it unreadable by other applications. However, using an auxiliary file, a sidecar store, risks separating the metadata from the file when it is copied. Windows uses two different approaches to storing metadata. The first is to use NTFS streams to store multiple data streams under a single file name. This is how Office document properties – author, word count and so on – are stored. This is a neat solution, but as soon as you try copying the file to a system that doesn’t implement NTFS streams the metadata is lost. The second method is to store the metadata within an OLE compound document. An OLE compound document can store multiple objects within a single file, and one of them can be the metadata. This works well, but it’s complicated and again a format that only Windows understands. An added twist to this story is that often, on an NTFS formatted disk, each object in an OLE compound document is stored in a separate NTFS stream!

To enable metadata to be stored in a more general way, manufacturers have resorted to modifying file formats in ways that they hope will be backward compatible. For example in the photographic world we have EXIF, IPTC, XMP and so on, all extensions to the well-known photo file formats (usually JPEG and TIFF). Of these the most commonly encountered is EXIF, as it has been adopted by most digital camera manufacturers, despite the absence of any standard or even a standards organisation to look after it. It may be common, but it has its limitations – it’s antiquated, fragile, inflexible and it doesn’t support international languages. It is generally agreed that XMP, an XML-based metadata system invented by Adobe, is probably the way of the future, because Windows Vista has standardised on it as part of its new Property System. When Vista writes metadata it uses XMP, but it also has to cope with the many other different ways that metadata is actually stored in files. The solution is to use metadata handlers, one for each file type, that “know” how to read the metadata and present it to the system in a single unified form.

The Vista Property System can be retrofitted to Windows XP by simply installing Windows Desktop Search 3.0 (downloadable from the Microsoft website). In an ideal world we would use the Property System, but at the moment there is no managed library that gives access to it, and there is very little documentation available. However, in addition to the new Property System we have the Windows Imaging Component (WIC), which is another attempt to rationalise the way that Windows works with bitmaps including metadata. The good news is that WIC is fully supported by a managed library – the not so good news is that this is part of the .NET 3.0 class library, and part of a whole new approach to graphics which isn’t in any way integrated with the .NET 2/GDI+ graphics classes, and uses the DirectX graphics engine.

If you think that this all sounds like an awful mess, this would be a fairly accurate summary of the situation. In the future the Vista Property System and whatever follows .NET 3.0 should make it all work in a logical and unified way, but for now it is difficult to be sure that we have found the best way to do anything with metadata or bitmap images.