Further thoughts on design

A few more thoughts on the design of the framework:

1. Logging (textual)
2. Logging (packets)
3. Diff'ing (packets)
4. Live history

Logging (textual)
The way that the textual output of the framework is delivered needs to be well-thought-out. While it's obvious that "test passed" / "test failed" is all that should be necessary for standard runs (INFO level), there are many distinct levels below INFO that will be necessary. A few of these off the top of my head:

• Full packet sent'd/recv'd over network IF
  This is obvious. Just a Packet sent/recv, with the BinHex print-out
  
  
• State-change
  This should somewhat reflect the diagram provided on Page 23 of RFC 793. This behavior will also make it easier to track issues in the flow.
  
  1. Closed
  2. Listen
  3. SYN sent
     
  4. SYN recvd
  5. Established
  6. Fin Wait-1/2
     
  7. Close Wait
     
  8. Closing
     
  9. Time Wait
     
  10. Closed
• Response generation
  Automated response generation (i.e. Send packet X, recv packet Y. Validate that Packet Y is what we were expecting, and then generate Packet Z that is a valid response/ACK for Packet Y).
  
  
• Field generation (dynamic)
  When sending information, some fields (like IP length, checksums) may be generated dynamically/on-the-fly.
  
  
• Manual field override
  Testing invalid states/data will also be necessary. Manually overriding fields (e.g. bad checksum) should provide extra insight. This might belong on the same logging level as Field Generation, but with a different prefix.

Logging (Packets)
Creating a TCPDUMP-formatted file with the contents of all of the packets from a test session will be invaluable in determining what, specifically, went wrong during the tests. This will make the use of third-party tools possible, and should complement the configurable logging well.

Differential Analysis
Simple packet replay, will only be useful in select circumstances. Ideally, sending packet X will always generate response Y (with some variables, such as timestamp). Playback of a TCPDUMPed session would allow this, but it would need a framework to determine what fields are important for validity, and which aren't. Since it would be difficult to require the specific lack of a particular response, this application may be very limited.

Live History
Having a history (in memory) of all of the packets sent/received during the current test will be a necessity. This is required to make sure that, for example, all of the packets were properly ack'ed as we continue sending packets. Also, it will be necessary to have internal state information about how much data has been ack'ed, and its correlation to the window size.

Read more...