Pointers on pointers

Back in the days when C was the language of choice, pointers meant programming and vice versa. Now in the more sophisticated and abstract days of C#, and even C++, raw pointers are a facility that is provided but not really encouraged.

In a modern language the argument is that you should never need to get down and dirty with pointers because to do so simply reveals that you are thinking at too primitive a level and in severe danger of meeting the real hardware that underpins everything. While this is true there are still times when reality pokes through the abstraction and you do have to interact with the hardware in ways that can only be achieved using pointers. In addition there are all those wonderful C and C++ programs that use pointers and need to be converted to something more polished and safe. In short, you certainly should know about pointers even if hopefully you never actually make use of them in a real production application. If you do find that pointers are essential to something low level that you are trying to implement then it is also important that you know how to implement them in as “safe” a way as possible.

Reference, pointer, address

First we need to clear up some confusion that exists between the three terms “reference”, “pointer” and “address”. Starting with the most basic “address” means the numerical address where something is stored. Most computer hardware assigns addresses using a simple incrementing scheme starting at some value and continuing to some other value, e.g. 0 to 1000, and each address corresponds to a memory location capable of storing some data with a given number of bits. This simple idea has become increasingly complicated as hardware has developed and with the introduction of hardware memory mapping. Now the address that data is stored at can change without the data being moved due to the use of address translation hardware. Even if the hardware doesn’t get in the way, addresses change while an application is running because the operating system or a garbage collection system often moves things around to make things more efficient. The point is that while once the address of something was a fixed and reliable way of accessing it, today it is surrounded by a range of problems.

The “pointer” is the step in abstracting the idea of an address. At its most basic a pointer is simply a variable that can be used to store the address of something. You can use a pointer to gain access to the data item pointed at – a process called dereferencing. You can also subject the pointer to arithmetic operations that move its location in the “store”. That is, if there are a number of objects stored one after the other in the store you can perform pointer arithmetic to change the object that is pointed at. Pointer arithmetic is the reason that many programmers like pointers but it is also the reason why pointers are dangerous. A mistake in the pointer computation can result in it pointing somewhere it shouldn’t and the whole system can crash as a result. There really is no reason why a pointer shouldn’t be abstracted away from the basic idea of an address, but in most cases pointers are just wrappers for machine addresses and this also raises the question of what happens if the system does something that changes the address of an object. More of this later.

Finally we reach the highest point of abstraction of the address idea in the form of a “reference”. A reference is just that – a reference to an item of data or an object. If this sounds like a pointer there is a sense in which this is true but the key idea is that you can’t manipulate a reference directly. That is, while there certainly is pointer arithmetic there can be no reference arithmetic. All you can do with a reference is to pass it to another user or dereference it and access the data that it references. As in the case of abstract pointers there is no reason why a reference shouldn’t be abstracted away from the underlying machine address but in most cases, and in C# in particular, a reference is just a wrapper for an address. In future implementations, however, a reference could be implemented as a handle to a table, to another table, to a resource index and so on, until finally the hardware converts the reference to the address of an actual data object. The point is that while we all know that a reference, and for that matter a pointer, in C# is simply a wrapper for an address this is a detail of implementation and not something you should rely on or make use of.

Pointers

In C# we make use of references all the time in the form of variables that have been assigned any reference type. For example, if you create an instance of a class then the instance variable isn’t an object but a reference to an object of the appropriate type. That is, after:

MyClass MyObject = new MyClass();

…then MyObject is a reference to an instance of MyClass. In practice it contains the address of the instance but as already explained you shouldn’t rely on this form of implementation. This storing of a reference in MyObject rather than a value is most clearly seen if you create another variable which references the same object as MyObject.

MyClass MyObject2 = MyObject;

Of course, we now don’t have another complete independent copy of the object referenced by MyObject, instead both variable reference the same instance. If you make a change to the instance using MyObject2 say then the same changes will be found via MyObject. The difference between a reference and a value basically comes down to this assignment semantics – does assignment produce a copy of the original data/object. If it does then we have value semantics; if it doesn’t we have reference semantics.

Pointers are a generalisation of a reference type to include pointer arithmetic. Pointers are so dangerous that they have to be quarantined within your code. First the entire project has to be marked as unsafe by using the Project Properties to set the Build, Allow Unsafe Code flag. Then any use of pointers has to be enclosed in an usafe{} block to mark it out even more clearly. Even more restrictive is the fact that you cannot create a pointer to anything, only to a restricted subset of types that have a simpler way of using memory so making pointer use slightly less tricky. Essentially you can only create a pointer to any simple value type, e.g. int, float, char, to an enum, to another pointer or to a struct that doesn’t contain other managed types. So you can’t have a pointer to an object, or to a delegate or to a reference. This is fairly restrictive and basically amounts to not allowing pointers to anything created on the heap or subject to dynamic memory management. However you can have a pointer to a struct that contains simple value types and you can create pointers to arrays of simple value types. You can also have a pointer of type void, i.e. a pointer to an unknown type but to be of any use in, i.e. to use pointer arithmetic, you need to cast a void pointer to a pointer to a given type.

To declare a pointer type C# uses the C++ like syntax:

type* variable;

The * is the dereferencing or indirection operator and is generally used in conjunction with the address of operator &, which as its name suggests, returns the address of a variable. For example:

unsafe
{
    int* pMyInt;
    int MyInt;
    pMyInt = &MyInt;
}

…creates a pointer to an integer, i.e. pMyInt and stores the address of the integer MyInt in the pointer. The first important thing to note is that a pointer does not inherit from an object and so there are no methods associated with it and no boxing and unboxing. For example, you can’t use a ToString() method call to display the value of a pointer. What you can do, however, is to use a cast to convert a pointer to a more usual data type. So, to display the contents of the pointer you would use something like:

MessageBox.Show(((
    int)pMyInt).ToString());

Of course this assumes that the current implementation of int is large enough to contain a pointer, i.e. a machine address.

The indirection operator returns the values stored at the address that a pointer is pointing at. For example:

MessageBox.Show((*pMyInt).ToString());

…displays the current contents of MyInt, i.e. the value of whatever pMyInt is pointing at.

The indirection operator and the address of operator really are the inverse of one another. That is:

MessageBox.Show((*&MyInt).ToString());

…just displays the content of MyInt. A pointer can be null and applying the indirection operator in this case generates an exception. Obviously it makes no sense to use indirection on a void pointer, what would the data type of the result be, but you can always cast a void pointer and then use indirection. Notice that this process can produce complete nonsense. For example, consider:

void* pMyData = &MyInt;
MessageBox.Show((*(
    double*)pMyData).ToString());

This sets a void pointer to an integer, i.e. a 32-bit integer, then casts it to a double pointer, i.e. double*, and finally uses the indirection operator to return the value so pointed at. If you try this out you will find that it works but it is mostly nonsense because the original int was only 4 bytes of storage and the double is 8 bytes. Where did the additional 4 bytes come from? The answer is that you have succeeded in reading data from a neighbouring memory location, one that you normally would not be able to access. Of course, reading from a memory location that you don’t understand is fairly safe, but who knows what the effect of writing to such a location is going to have. For example, try:

int MyInt2 = 0;
int MyInt=1234;
void* pMyData = &MyInt;
*(double*)pMyData = 123456.789;
MessageBox.Show(MyInt2.ToString());

You might be surprised to discover that the value of MyInt2 has changed and is no longer zero, even though it isn’t assigned a new value within the program. The simple explanation is that MyInt2 is allocated storage alongside MyInt and when we assign an 8-byte value to MyInt the extra 4 bytes overwrite MyInt2. This is clearly dangerous, unexpected and usually unwanted behaviour and it is in this sense that the code is “unsafe”.

One very common use of pointers is to get at the internals of a data type. For example, suppose you want to retrieve the four bytes that make up a 32-bit integer:

int MyInt = 123456789;

We can always use a void pointer to get the address of any variable:

void* MyPointer;
MyPointer = &MyInt;

Then we can cast it to a pointer to any of the standard types and use pointer arithmetic, in this case to a byte which is then converted to a char:

byte* pMyByte = (byte*)MyPointer;
char MyChar = (char)*(pMyByte + 3);
MessageBox.Show(MyChar.ToString());

The reason we don’t go directly to a char pointer is that a char is two bytes in size and we are converting a 4-byte int to four ASCII rather than Unicode characters.

In most cases there are managed ways of gaining access to the internal structure of the common data types using either the Convert or BitConvertor classes. In this case the BitConvertors GetBytes method can be used to convert the int to a byte array and then any of the bytes can be converted to a char using the Convert class:

Byte[] Bytes = 
    BitConverter.GetBytes(MyInt);
MyChar = Convert.ToChar(Bytes[3]);

As long as there is a GetBytes method that will convert the data type into a byte array you don’t need to use pointers.

Multiple indirection

If you think you have got the idea of indirection then it’s time to put you to the test. The real mark of a pointer expert is being able to handle double, treble and more… indirection. In theory this is easy. For example:

int** ppMyInt;
int* pMyInt;
int MyInt=1234;
pMyInt = &MyInt;
ppMyInt = &pMyInt;
MessageBox.Show((**ppMyInt).ToString());

In this case we declare a pointer to a pointer, i.e. ppMyInt and use it to point at pMyInt. To display the value pointed at by pMyInt, i.e. the value in MyInt, we have to use two levels of indirection as in **ppMyInt. In this case double indirection is fairly easy to follow but in real cases it can become very difficult to work out when you need a pointer or a pointer to a pointer and so on.

Pointers, arrays and fixed

There is a very close relationship between pointers and arrays, indeed you could say that pointers in languages such as C and C++ were introduced just so that it was possible to create arrays. In principle the address of the first element of an array can be used to find any element of an array but things are a little more complicated. For example,

int[] MyArray = new int[10];
for (int i = 0; i < 10; i++)
    MyArray[i] = i;

int* pMyArray = &MyArray[0];
MessageBox.Show((*pMyArray).ToString());

This should create a pointer to the first element of the array but if you try it you will discover that you simply get an error to the effect that you can’t take the address of an unfixed expression. The reason is that while MyArray[0] is just an integer variable the compiler knows that the array is a managed object and can be moved at any time. If you were to take the address of an array and then it moved the address would be useless. To take a meaningful array address you have to use the fixed keyword:

fixed(pointer declaration)
{
    instructions to be carried
    out while data is fixed
}

For example:

fixed (int* pMyArray= &MyArray[0])
{
    MessageBox.Show((
    	*pMyArray).ToString());
}

…will work and display the contents of the first element of the array. You can also use the array name as a shorthand for &MyArray[0] as in:

fixed (int* pMyArray= MyArray)
{
    MessageBox.Show((
    	*pMyArray).ToString());
}

Notice that the pointer declared in the fixed statement goes out of scope when it ends so you can’t use pMyArray unless the array is fixed.

If you want to access other array elements then you simply use pointer arithmetic as in:

fixed (int* pMyArray= MyArray)
{
    MessageBox.Show((
    	*pMyArray+5).ToString());
}

…which displays MyArray[5]. Notice that this is rather more subtle than you might think as adding 5 to the address of the start of the array actually adds 5 times the size of a single integer element. That is, the arithmetic operators have been overloaded to add units in the size of the data type being pointed to. There is a sizeof operator which returns the size of any value type and this is used to work out what to add to a pointer. That is, pointer+5 is translated to pointer+5*sizeof(pointertype).

To complete the connection between arrays and pointers you can also use array indexing as a shortcut to dereferencing and pointer arithmetic. That is pointer[5] is a synonym for *pointer+5:

fixed (int* pMyArray = &MyArray[0])
{
    MessageBox.Show(
    	pMyArray[5].ToString());
}

There is a restriction on the use of the fixed pointer in that it cannot be modified within the fixed statement. This isn’t a problem as you can simply make a copy of it:

fixed (int* pMyArray = MyArray)
{
    int* ptemp = pMyArray;
    MessageBox.Show((
    	*++ptemp).ToString());
}

…which displays the contents of MyArray[1].

This sort of pointer manipulator works with multidimensional arrays. For example:

int[,] MyArray = new int[10,10];

fixed (int* pMyArray= &MyArray[0,0])
{
    for (int i = 0; i < 100; i++)
    {
    	*(pMyArray + i) = i;
    }
}

This initialises a two-dimensional array by accessing it as a linear block of memory. Of course, the order in which the array is initialised depends on how it is stored in memory and I leave it to you to breakpoint the example and discover if it is stored in row or column order. It is this sort of trick, i.e. accessing a 2D array as if it was a 1D structure, that made (and still makes to some) pointers so attractive.

Notice that you can initialise multiple pointers within a fixed statement as long as they are all of the same type. To initialise multiple pointers of different types you have to use nested fixed statements one for each type.

Structs

It is time now to turn our attention to structs and pointers to structs. You might well imagine that if an array is something you have to fix before using pointers to it then you would certainly have to fix a struct in the same way. You don’t have to because a struct is a value type and allocated on the stack. So you can use:

public struct MyStructType
{
    public int x;
    public int y;
};
MyStructType MyStruct=new MyStructType();
MyStructType* pMyStruct = &MyStruct;

Now how do you access a field using a pointer? You can use the fairly obvious:

(*pMyStruct).x = 1;

That is, dereference the pointer and use the usual dot selector. However in a homage to the C++ usage you can also write:

pMyStruct->y = 2;
MessageBox.Show(pMyStruct->x.ToString());

That is, the -> dereferences the pointer and selects the field in one go.

Strings

It is usually said that you can’t have a pointer to a string because a string is a managed object but, just like an array, you can fix a string and then you can initialise a char pointer to the first character in the string. For example:

string MyString = "Hello pointer world";
fixed (char* pMyString = MyString)
{
    MessageBox.Show((*(
    	pMyString + 6)).ToString());
}

…creates a string in the usual way, fixes it and obtains a char pointer to its first char. Then we can perform pointer arithmetic to access the 6th character in the string.

Memory allocation

As well as working with value types you can create your own primitive data types using the stack. The statement stackalloc type[n] allocates enough stack to store n copies of the stated data type and returns a pointer to the start of the allocation. You don’t need to fix the storage as the stack isn’t moved or garbage collected while the variables are in scope. You also don’t have to remember to deallocate the memory because the stack is automatically cleaned up when the variables go out of scope, usually when the method that declared them returns. For example:

int* pMyArray = stackalloc int[100];
pMyArray[10] = 34;
MessageBox.Show(pMyArray[10].ToString());

…allocates 100 integers, i.e. 400 bytes, on the stack and uses the pointer to store 34 in the four bytes starting at the 40th byte and then displays this value.

Notice that the use of the array indexing makes this look exactly like allocating and using a standard array. However, the block of memory really is just a block of memory that you can do what you like with. For example:

public struct MyStructType
{
    public int x;
    public int y;
};
MyStructType* pMyDataBlock =
    stackalloc MyStructType[1];
pMyDataBlock->x = 34;

…allocates a struct on the stack but this is really just sizeof(MyStructType) which happens to be just two integers, i.e. 8 bytes.

We can use the pointer to the structure in the usual way to set or access a field. That is, we can use the block as if it was a struct but if we want to we can just treat it as a block of 8 bytes and use it as some other data structure. For example, if you want to treat the data as an array of int you can by casting the pointer:

*((int*)pMyDataBlock) = 36;

This expression is a little difficult to follow so a simpler multi-step version is:

int* pMyArray = (int*)pMyDataBlock;
pMyArray[0] = 36;

This casts a pointer to int then uses array indexing to access the first element of the array which is the same as the x field of the struct.

Notice that this correspondence depends on the way the struct is organised in memory and currently the C# compiler stores fields in the order that they are declared. A range of attributes can be applied to ask the compiler to use particular memory layouts for a struct, see Mastering structs in C#, but even then you are making use of details of implementation that could change and make your program invalid.

What’s it all for?

You might be excited to learn all about pointers but, as you might have detected, they are not really a good idea. C and C++ programmers moving to C# tend to think in terms of pointers and hence like the facility, but in practice you almost never need them. If you need to do a quick conversion of a C/C++ program that uses pointer then it might be acceptable to use C# pointers to get things moving but it should be considered a stop-gap measure until you can implement the ideas without pointers.

About the only time pointers might be necessary is in making use of API calls within the P/Invoke subsystem. Even here there are usually alternatives and often passing by reference solves even pointer to pointer problems. Things do get more difficult when you need to pass a pointer to a block of memory and in this case you might have to use a fixed array or allocate memory on the stack and pass it directly to the API call.