[I)ruid] from BreakingPoint Labs has been doing quite a bit of protocol reverse engineering as part of his work. He put together a post covering some of the tools that have been useful for this task. Text-based protocols have a lot of human readable characters that can help you identify fields. Binary protocols don’t have this luxury though. He recommends the Protocol Informatics Project for tackling these situations. It applies bioinformatics algorithms to network traffic. You give it a packet dump of the protocol and it compares them to find similarities the same way genetic sequences are compared. It can be confused by protocols that waste a lot of space, but it’s still a very clever approach to reversing.