Computer Othello, Part 5: Resources

This post lists the best info I could find on how to write a computer version of Othello.

General

http://en.wikipedia.org/wiki/Reversi#Rules
Rules for Othello/Reversi

http://chessprogramming.wikispaces.com/Othello
Details on bitboards, hashes, deep-first searches and transposition tables. Contains source code (generally in C or C++)

http://www.radagast.se/othello/howto.html
A description of what is required to implement a better than average computer player.

https://skatgame.net/mburo/ps/compoth.pdf
A paper that describes the best Othello computer players from the 80s and 90s (IAGO, BILL and LOGISTELLO)

Strategy (human and computer)

http://radagast.se/othello/Help/strategy.html
http://www.samsoft.org.uk/reversi/strategy.htm

Implementation

http://users.informatik.uni-halle.de/~jopsi/dass4/
A break-down of the tasks involved in creating an Othello game. Has info on implementing the rules and how the minimax search works.

http://www.dcs.gla.ac.uk/~daw/masters-projects/dissertations/Colquhoun.2008.pdf
Computer science student's paper.

http://www.cs.kent.edu/~jmelnyk/othello/
A description of someone's attempts to write-up Othello depth-first searches (alpha-beta, negascout, MTD(f), Multi-ProbCut, etc.)

Computer player

http://samsoft.org.uk/reversi/openings.htm
A list of the standard Othello openings.

http://xenon.stanford.edu/~lswartz/cs221/desdemona_writeup.pdf
Description of various evaluation strategies.

http://othello.dk/book/index.php/Thor_Database
The Thor database was the only archive of Othello games that I could find on the net.

Finally

If you want to see what I did with this information, there is my source code for Othello. It's fairly well written, the UI looks okay and is easy to use and the computer player plays well. I implemented most things you'd do in a world-class computer player. However, you'd have to make it a lot more efficient, if you wanted to take on those players.

It was interesting, frustrating and fun to try to write a decent Othello game. I learnt a huge amount too.

Troubles with Unity and Mono

Using Unity for my Othello game was not without issues. The biggest issue I experienced related to Unity's use of Mono. Here is why:

A depth-first search generates a lot data. Oridinally, in .NET, that would be fine. You create the data, process it and forget about it - the .NET garbage collector (GC) will release the data from memory when required. That isn't true of Mono 2.6. The garbage collector in Mono 2.6 is kind of rubbish. (See what I did there?) Mono 2.8 has a new garbage collector, but Unity 3.5 uses Mono 2.6. And my Othello game uses Unity 3.5.

For my Othello implementation, every turn by the computer player was a new search. It leaked memory all over the place because the GC wasn't dumping the data. It could easily use a gig of RAM over a game, even with shallow searches (depths of 5 or less).

To resolve this issue, I - though Andrew came up with the idea - used a struct instead of a class for the objects used in the search (EvaluationNode and GameState). I re-used memory by storing search results in arrays where the indexes were reset every turn of the game. This negated the need for garbage collection. In the source code, the classes that manage this process are called the EvaluationNodeBuffer and EvaluationNodeCollection.

These changes turned out to be a really good use and re-use of memory. It is also an excellent example to demonstrate the differences between a struct and a class. It also allowed me to search to much greater depths for the computer player.

A problem with this technique is that it makes it very difficult to write code to re-use part of the search tree between turns. Finding which parts of the tree to prune and to then re-organise the arrays and indexes would be technically tricky and CPU intensive. Therefore, for now, the computer player continues to re-searche all game states between turns. Furthermore, any sort of parallel programming to speed-up the search would be hindered by this approach.

What about Unity 4? That's out soon. Will that support the newer version of Mono? Unfortunately not.

We will be shipping Mono 2.6 with Unity 4.0. This will allow the same subsets of .NET features as in Unity 3, depending on the profile you choose in your player settings. (Unity 4 FAQ)

Computer Othello, Part 4: Trials and Transposition Tables

By the time I implemented transposition tables for my computer player in Othello, I'd finished most of the components of what's required for a decent Othello computer player. I didn't really need to add a transposition table, but I really wanted to understand and implement all the aspects of a computer player for these sorts of games.

A transposition table is used to help speed-up a depth-first search. It does this by keeping track of game states that it has already searched. Along with the game state, it records the value it found for that state in the evaluation function. (Therefore, a suitable data-type for a transposition table in C# is a Dictionary<gamestate,float>. GameState overrides the GetHashCode method to provide an unique as possible hash of the game state. The float holds the value from the evaluation function.) If the search finds the same game state again (via a different path), it uses the value in the transposition table rather than re-calculating it. This saves time.

Intuitively, I wouldn't have thought Othello would have many ways in which different sequences of plays could result in the same game state. Any reading about computer Othello says otherwise, however. In practice, I managed to wipe a couple seconds off a search by using a transposition table.

When I began writing my transposition table I looked for a good way to hash a game state. I found a description of the Zobrist hash:

One useful method for hashing positions in games like Othello and Chess is Zobrist hashing [12]. A Zobrist hash consists of an XOR sum of several bitstrings. For each square on the board, there is one randomly generated bitstring representing a black piece and another representing a white piece. A position's Zobrist hash is formed by XORing together the appropriate bitstrings. The primary benefit of Zobrist hashing is that it can be incrementally updated very quickly by XORing it with the bitstrings for the pieces that have changed. Zobrist hashes also have the advantage of uniform distribution. (Applications of Arti ficial Intelligence and Machine Learning in Othello)

I implemented a Zobrist Hash for Othello, created a dictionary of type Dictionary<ulong,float>, then wondered why I was getting so many bad results. The problem? When you add a key to a dictionary it'll call GetHashCode on the object to retrieve the hash. What happens when you add a hash as a key to a dictionary? The same thing. I was double hashing! Not only that, I was going from a 64-bit Zobrist hash to a 32-bit .NET hash. I was burning time in creating the hash as well as losing information and most likely increasing the number of collisions between different game states. All pretty ugly stuff. The solution, outlined above, was very simple: ditch the Zobrist hash, override the GameState GetHashCode to have:

        public override int GetHashCode()
        {
            return (PlayerPieces | OpponentPieces).GetHashCode();
        }

and change the dictionary to have GameState as the key. The result isn't as fast as if I had implemented my own type of hashtable with my own hash type, but it was a lot easier to do.

In the end, adding a transposition table to my game was relatively easy to do, though I took a long and unnecessary detour to achieve it.

Computer Othello, Part 3: Othello computer opponent

After working on the game archive stats, I was ready to create a computer player. Previously, the computer player had played randomly (even Andrew was able to beat it!) To create a computer opponent, I needed to do two things; create an evaluation method and plug-in a search algorithm to find the best estimated play to make.

Evaluation

One of the trickiest things to do with the computer opponent is to evaluate the current game state. Given any arrangement of pieces, you need to be able to determine which of the possible plays is the best. You need an heuristic to do this. An heuristic is an estimation of what you think is a good play to make. The factors that I used to evaluate the state of the game were:

* Number of pieces
* Number of playable squares (usually called "mobility")
* Number of empty squares next to the player's piece ("the frontier", or "potential mobility")
* Various corner and edge patterns (edges and corners are usually better moves)

Pieces and mobility

The most naive heuristic is to count the number of pieces each side has. Ultimately, this is the only measure, piece count determines the winner. However, until the end game, it has little to do with who is going to win.

A less naive approach is to count how many plays can be made. A state where 7 plays are possible is almost certainly better than one where you have to skip your turn because you can't make a play. I had already calculated where a player could place a piece in their turn as part of the user interface. Therefore, I got the mobility measure for nothing.

Finding the empty squares next to a player's pieces was also relatively simple to implement. It was little more than a variation on the mobility algorithm.

Patterns

Image that elucidates where the X and
C-Squares are on an Othello board.

Implementing the patterns was relatively easy to do but it was one of the last additions to the evaluation function that I made. I had previously dismissed this approach as too simplistic to provide much of an improvement to the evaluation. I couldn't have been more wrong. Once I had coded the patterns, the computer player beat me easily. I've since beaten it once or twice, but it went from a disgustingly below average player to a better than average player.

The patterns that I check for are:

• Corners (a good play for the current player)
• X-Squares (bad play)
• C-Squares (bad)
• Corners and X-Squares (good)
• Corners and C-Squares (good)
• Edges (good)

Furthermore, I check for lots of combinations of these patterns. I.e., not only does a game state get points for a play on a corner, but gets even more points for the multiple corners. I didn't check to see if this approach improved play, but I suspect it would.

Piece stability and parity

There were a couple of major components of a good Othello evaluation function that I didn't include. These were piece stability and parity. Piece stability, i.e., finding the pieces that cannot be flipped, is one of the trickier things to determine. There is a good description of how to do it here. I couldn't think of a really efficient way to implement stability, so I left it out.

Parity, i.e., determining who plays last, was relatively simple to implement in its basic form. By default, white will always play last, and therefore has an advantage. For black to play last, someone has to miss a turn. The basic approach to parity didn't really seem to impact the performance of the computer player, so I left it out of the evaluation. A sophisticated form of parity - one where isolated parts of the board are evaluated for parity (an isolated section is one that is surrounded by pieces and edges) - seemed too tricky to implement, so I never tried.

Depth-First Search

I took my negamax, alpha-beta pruning negamax and negascout search methods from my noughts and crosses source code and adapted it to work for Othello. That was fairly easy to do, although my original code was a bit rubbish.

Initially, I thought I'd use negascout for Othello as it is the best of the three. However, for it to work effectively (i.e., better than an alpha-beta pruning negamax), it needs to do shallow searches of the game tree, or find some other way to have a pretty good attempt at pre-ordering the plays from best to worst. Negascout generally does a mini-search within a normal negamax search. It was a more involved task than I suspected. Once I had implemented the patterns approach to the evaluation function, I realised that my computer player was pretty good. Therefore, I decided not to pursue a negascout algorithm for Othello.

Opening book

With all the work that I did to be able to display the history of games to the human player (percentage of games that made a play, percentage of those where black wins), I was serendipitously writing the code for the computer player. The computer opponent uses this information in a similar way as a human.

End game search

One of the things that I didn't do for the computer player was an put any work into an end-game search. This sort of search is much deeper and tries to get search until the end of the game. Once a computer has this information, it'll know if it has won and exactly which moves to make to ensure victory. Until the end-game search, all other plays are calculated guessing. All I did to the computer player approaching end game was increase the search depth.

Conclusion

I completely underestimated how difficult it would be to create a competent computer player. I now have a much greater respect for people who managed to create computer players that are vastly superior to mine on machines that are vastly inferior to today's technology. It's true that .NET is not really up to the challenge (unmanged code like C and assembler would be much more suitable), but you'd think a modern processor using .NET could get close to Pentium using C. From my experience, it didn't. But in truth, it was the algorithms that weren't good enough. I would have many more things to do to be able to compete with other computer Othello players. E.g., stability detection, parity, better potential mobility, negascout, Multi-ProCut, end-game solving, much deeper searching, training and machine learning etc. There is an immensity of improvements that I didn't even touch on. I'm not unhappy with what I achieved, more very impressed that people have done so much better.

Computer Othello, Part 2: Game archive as trainer

I wanted to create a computer game of Othello that did more than simply defeat the human player. I thought of trying to provide a story to the player to help them learn and improve against others. One idea was to process the Thor archive and create stats to apply to each potential play. In that way, the player can see how many people have played the position in the past and how well they did. However, with 100,000 games in the archive, doing this efficiently was tricky.

Initially, I tried using a trie data structure. It seemed like an appropriate structure. However, performance was poor and programming against it wasn't overly intuitive. I tried changing how the data was stored to try to improve performance. Instead of using a human readable notation (i.e., algebraic notation, such as "E6"), I stored each play as a single Unicode character. With 60 possible plays, I used the alphanumeric set, capital letters and symbols. I did this by simply taking a numeric board position (0 to 63), adding 48 to it (to move it into the alphanumeric area of Unicode) and converting it to a char type. This improved things, but performance using the trie was still terrible.

I ditched the trie and wrote LINQ statements against a list of strings. Performance improved, but it was still poor. I switched the search from LINQ to use the BinarySearch method of a list. For a binary search to work, I had to order the data. I was surprised to find that the LINQ OrderBy method didn't sort my data the way I needed. I suspect that it doesn't distinguish case and can't cope with unusual characters. I switched to Array.Sort. Even that didn't work without the StringComparer.Ordinal option. I also needed StringComparer.Ordinal option for the BinarySearch method. All interesting little hurdles.

After all that (plus some more optimisations discovered by using Performance Analyzer in Visual Studio), performance was massively improved but still not perfect. I decided to process the calculations on a separate thread so that it didn't hang the user interface.

I was done. Stats appeared either instantly or within half a second and either way the user interface was not impacted. Yet, I still wasn't done!

Othello is a highly symmetrical game. The first play, a choice of four possible positions, is symmetrically equivalent. No matter where black plays on the first move, one could rotate the board and the board would look the same as any other play. The people who made the Thor database understood this. That's why they standardised the archive to have E6 as the first play for all games. I needed to display the same info on the C4, D3 and F5 tiles that I displayed on E6. Otherwise, the less knowledgeable player may wonder why 100% of games are played at E6.

To be able to display stats on a non-standardised position, I needed to test for symmetry in the game board. I found methods to flip and rotate bitboards. That solved half the problem. I could check if one game state was the same as another. Next, I rotated the stats from a standardised to a non-standardised board. I generated lookup dictionaries to be able to rotate the indices of the board positions. For example, position E6 would become position C4 on a rotated board.

Display of game stats for black.
The blue highlighted square displays black played there in 0.027% of games and won 10% of those games.

After all that, I had a well working set of stats that I could display to the player to assist them in learning where the best opening moves are. It doesn't really provide any sort of story for the player, but at least it gives an indication of where a good play might be.

Later, I also used this data to help the computer player make its opening moves. But that's for another day.

GitHub for Windows

I'm tired of using Skydrive as a "source code control" solution. That's no solution, just a file back-up. What about versioning and sharing code?! I've used lots of revision control systems before. Here's the list:

• Visual SourceSafe
• SourceGear Vault
• Subversion
• TFS
• Mercurial

They've all been adequate but frustrating. The Internet is abuzz with talk of Git and GitHub. I tried setting up a GitHub account last month. After the first few steps, I looked at the long list of remaining tasks and decided it was "too much work."

To my rescue comes GitHub for Windows. Not only did it create my repository in seconds, it filtered out all the useless gunk that I don't want going into the code repository, namely binaries, ReSharper and Visual Studio profiler files. It still had some NCrunch files to ignore, but that's a minor issue. Could this be my revision control Holy Grail?

My Othello repository can be found at: https://github.com/ledpup/Othello.

Computer Reversi, Part 1.5: The Thor database (Othello tournament files)

As part of my research into Reversi, I looked for an archive of games. The only substantial one I found is called the Thor Database. It's an archive of Othello games by French archivers. They have archived are over 100,000 games. You can download the game databases here.

My initial motivation for finding an archive was to have a easy way to unit test my code to ensure it implemented the Reversi rules correctly. If the game engine could play through 100,000 games and calculate the final score, that's a good indication that the rules have been implemented correctly. An archive can also be used for book openings. I have another idea for the archive too, which I will hopefully reveal soon.

My Reversi source code has a Thor project with a bunch of methods and classes for extracting the data for whatever nefarious needs you may have. I don't think there is any code in C# on the net for this, so maybe one day it'll be helpful for someone. Maybe. (Probably not.)

The file format documentation is challenging to understand because it is written in French. I have translated (thanks to Nuz and google translate) the core elements of the document in the section below. The aspects I found challenging were:

• Interpreting the meaning of the black player's score (I needed to calculate the score as the final check to ensure my rules were working)
• Interpreting the way they recorded the individual plays
• Converting elements of a byte-array to a 16-bit integer. (The data are stored as little-endian. One way: BitConverter.ToInt16(new [] { thorArray[i], thorArray[i + 1] }, 0).)

To calculate the Black Score:

• Get sum of black pieces and the sum of white pieces;
• Whoever has the most pieces wins;
• The winner adds to the sum the number of empty squares;

E.g., Black has 44 pieces, White has 8 pieces; Black's score is 56. Or, Black has 13 pieces, White has 35; Black's score is 13. In the case of a draw, Black's score is 32 (as the empty squares are shared between Black and White, 32 is the only possible score for a draw). I still haven't figured out what it means for Black's score when the game ends before neither player can play a piece. The score seems inconsistent. This only occurs in 406 of 107,473 games. (Not a big deal.)

---

The data, such as Word and Longint, are stored in Intel format, ie the lowest byte first. There is a game file for every year. Games are stored in any order, but normally grouped by tournament.

Database header fields
All files in the database Wthor have a header of 16 bytes, followed by a number of game records all having the same size. The header consists of the following fields:

• Century file was created
• Year file was created
• Month file was created
• Date file was created
• Number of records N1
• Number of records N2
• Year of game
• Parameter P1: size game board
• P2 parameter: type of games
• Parameter P3: depth
• X1 (reserved)

Game fields

• Tournament Number
• Number of Black player
• Number of White player
• Number of black pieces (true score)
• Theoretical score
• Move list 

The plays are stored in chronological order. Row number (1-8) and column (A-H) can be derived from the following operation: column + (10 * row). E.g., a1 = 11, h1 = 18, a8 = 81, h8 = 88.

Tournament file

Each record (26 bytes) is an array of characters terminated by a binary zero. The effective length is 25 characters. There is a 16 byte header for this file. 

Player file

Each record (20 bytes) is an array of characters terminated by a binary zero. The effective length is 19 characters. There is a 16 byte header for this file.

Computer Reversi, Part 1 (Or how I learned to stop worrying and love the bits)

After noughts and crosses I thought I’d try Reversi (Othello). Reversi has simpler rules than chess but remains complex enough that it hasn’t been mathematically solved, yet.

I thought I’d find a wealth of information on the net about how to program a Reversi game. However, there wasn’t all that much out there. I found some helpful blog entries at Red Alt Blog. I used that as my starting point.

The best resource for understanding how to program Reversi is at the chess programming wiki. They have an Othello page as well as pages that helped me do a bitboard version of Reversi, with move generation and resolution (dumb7fill), population count (i.e., number of bits on the board) and board serialisation (using bitscans). I'm hoping it'll also be helpful for position evaluation later on.

The tasks needed to code a Reversi computer game with a learning computer opponent are

1. Game state representation
2. Move generation
3. Move resolution
4. Determine game over and winner
5. Graphic User Interface
6. Save/load and undo/redo moves
7. Position evaluation (i.e., fitness/objective function)
8. Depth-first search algorithm (e.g., mini-max or nega-scout)
9. Book/database based analysis (e.g., opening book, transposition tables) 
10. Mathematical optimisation (e.g., simulated annealing or genetic algorithm)

This blog entry addresses steps 1-6, providing source code for a Unity implementation of Othello. The game logic is written as C# and compiles as mono inside Unity. I assume knowledge of:

• C#
• Unity 3D
• Reversi rules

The following sections describe some interesting parts of the source code, relating with the first six steps outlined above.

Reading the source code

The entry point for the application (the Main method) is the Start method of the GameBehaviour.cs file. This file is attached to the main camera in Unity. The GameBehaviour.cs file creates the board and pieces as 3D objects and hosts an instance of GameManager (see below). It also manages the UI controls to save/load games, undo/redo moves, setup the human/computer players, start a new game, etc.

The files that do all the real work are:

• GameBehaviou.cs
• GameManager.cs
• GameState.cs
• BitBoardHelper.cs
• Play.cs

There are source files, 64-bit Windows binaries and Mac OSX binaries. There are two solutions files contained in the source files. The one named with VS2010 is the one you should open in Visual Studio. It contains a test project and a preliminary attempt at reading the Thor database format. But that will be for another post. You can load the project in Unity via the "Reversi.unity" in the Assets sub-folder.

GameManager

The GameManager class does the work-a-day tasks of the game. Tasks such as:

• Save/load
• Undo/redo
• Tells you whose turn it is
• Tracks the list of plays
• Tracks the turn number

GameState

My GameState class handles most of the game rules of a Reversi game. You pass it two unsigned 64-bit integers (ulong in C#). These numbers are a bitboard representation of the game. The first number represents the current player's pieces. The second number represents the opponent's pieces. E.g., the starting position for black is represented as 0000000000000000000000000001000000001000000000000000000000000000 in binary. When you lay out those zeros and ones on an Reversi board, you get:

Why represent the pieces like this? Bitboards appear to have advantages over other board representations. They appealed to me because I often feel withdrawn from the goings on of the computer. Bitboards were a chance to get in closer to the CPU and play around with individual bits.

One interesting aspect of the GameState class is that it is temporally agnostic. It doesn't know what turn of the game it is. It doesn't even know whose colour is to play next. All it cares about is that it is someone's turn and it figures out where they can play and what happens to the board once they do. I like the simplicity of not needing to deal with time.

The GameState class will tell you if the game is over and whether the current player has won.

BitBoardHelper

I created a BitBoardHelper class to add some extension methods to my ulong bitboards. These methods allow me to count the number of bits, find the indices of the bits and find the unoccupied bits of any two bitboards. I'll probably expand this class when I work on the computer opponent.

For finding the indices (bitboard serialisation) I used the De Bruijn method to do the bitscanning. For the bit count (population count) I used the SWAR Popcount routine.

Play

There is a Play class in the project that will find all the valid moves for the current player and also resolve a chosen move (i.e., flip of the pieces that need to flipped once a piece has been played). The basic idea is to focus one direction (cardinal or ordinal) at a time, scanning 8 locations in parallel. Red Alt explains it well (see under the bitboard heading). Once I could detect where a player could place a piece, I used a very similar method to resolve that move. This class is called by the GameState class and is only different to the BitBoardHelper class in that it has specifically Reversi functionality.

Graphic User Interface

The GUI uses the Unity 3D graphics engine. Much of the code that I use in the Reversi project is a simpler version to what is found in a previous article on Unity and path-finding. Unity displays the board and captures user input.

Save/load games and undo/redo moves

Save/load and undo/redo are interrelated. To do them, I need to keep track of the moves made by the players. If I do that, I can:

• Save: Write the move list to the hard drive.
• Load: Read the move list from the hard drive, start a new game, apply move list - one move at a time - until the last move is applied.
• Undo: Start a new game, apply move list until the desired turn (similar to load).
• Redo: Identical to undo.

Part 2

In Part 2, I intend to cover the aspects of creating a computer opponent for Reversi. The primary goal is to be able create an opponent that can beat me in a game. (Not a highly goal.) Hopefully, I'll be able to have it beat all but the best Reversi players.