Sunday, May 31, 2009

Logic Validation

Looking for some input on folks familiar with low-level TCP. I've got some general questions about proper values of Sequence # and Ack # over the course of a TCP session.

ACKing Packets

A segment on the retransmission queue is fully acknowledged if the sum of its sequence number and length is less or equal than the acknowledgment value in the incoming segment.

A packet is received with sequence number "seqn" and TCP payload with "len" octets of data. The receiving party should compare "seqn" to the variable outlined in the RFC, "rcv.nxt". If they match, "" is incremented by "len".

The ACK number of outgoing packets is always sent to ''-1.

What is done with packets received out-of-order, but that are entirely valid? Buffer them, then check each one's SEQ number when rcv.nxt is incremented?

Delayed ACK response
This also raises the question of how to properly ACK a received message, when packets aren't already being set. In some instances, it may be necessary to send an empty packet with the ACK flag set. How to determine when this needs to be done?


IP and Port Representation

Small hurdle that I tried to tackle tonight is the representation of IP addresses and ports. There are at least 4 representations of each that will need to be dealt with regularly:

- ASCII (e.g. "", "80")
- Host-byte-order integer, useful for ports (16777343, 80)
- Network-byte-order integer, used in raw packets (2130706433, 20480)
- Network-byte-order byte-string, used by "inet_aton" and the like. ('\x7f\x00\x00\x01', '\x00\x00\x00P')

So, the IP address and Port were encapsulated in classes. This should make dealing with them slightly easier.



I was having some issues earlier wrapping my mind around why, exactly, packing strings didn't give me a string with the "\x##\x##\x##" format. Evidently, that is only necessary for characters that do not have a normal, printable equivalent.


>>> "\x7D" + "\x7B"

Anyway, the following output is what had me confused, and why I spent about an hour or so now looking into the 'struct' package and why it wasn't giving me what I thought I wanted. Notice the plaintext "test" at the end. It seemed out of place, so I was assuming that I was doing something wrong.

>>> from pcsextension.payload import payload
>>> from pcs.packets.tcp import tcp
>>> p = payload('test')
>>> t = tcp()
>>> = p
>>> t.chain().bytes


Rejected Email


If you're reading this, I'm having some issues contacting you via email:

This is an automatically generated Delivery Status Notification



Delivery to the following recipient has been delayed:

Message will be retried for 2 more day(s)

Technical details of temporary failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 451 451 4.3.0 Message held for human verification before permitting delivery. For help, please quote incident ID 13393043. (state 18).


Monday, May 25, 2009

Memorial Day

Happy Memorial Day, All!


Sunday, May 24, 2009

Logging Framework Change

Decided I was going about it all wrong (it was getting late). Changed the implementation to simply provide convenience methods on top of the Python 'logging' package's framework -- with the added benefit of not having to worry about the nitty-gritty of the 'logging' package. Just simple, direct access to the log methods.

>>> from loggable import Loggable
>>> class A(object):
... def __init__(self):
... self.log = Loggable (self)
>>> a = A()
2009-05-24 12:48:08,654 - __main__.A - INFO - test
>>> a.log.critical('test')
2009-05-24 12:48:19,475 - __main__.A - CRITICAL - test
>>> a.log.state('test')
2009-05-24 12:48:22,978 - __main__.A - STATE_CHANGE - test
>>> a.log.pktsent ('test')
2009-05-24 12:48:26,402 - __main__.A - PACKET_SENT - test
>>> a.log.generated('test')
2009-05-24 12:48:49,516 - __main__.A - RESPONSE_GEN - test
>>> a.log.field('test')
2009-05-24 12:48:52,475 - __main__.A - FIELD_CHANGE - test

Also have been working on a simple TCP State Machine today. My theory is that we will need to know [1] what state the local TCP connection is in, and [2] what state the remote TCP connection should be in. A simple state machine should allow us to do this with some flexibility. Here is the base class/interface. Suggestions are highly welcomed:

class  TcpState(object):
Base TCP State class. Note that it will, by design, ignore all non-TCP layers.
The TCP State classes will not perform verification that the packets are all
to/from the same hosts. That filtering will be implemented elsewhere in the

def __init__(self):
self._nextState = None

def packetSent(self, tcpPacket):
Hand off a packet to the TcpState object, where the object will perform
any necessary calculation, and determine if a state change should be
caused by the provided packet.

The packet should be a packet that is considered 'sent' from
the TCP stack that this state belongs to.

@return True if the packet causes a state change. False otherwise.
return False

def packetRecvd(self, tcpPacket):
Same as packetSent, but in the opposite direction.

The packet should be a packet that is considered 'received' from
the remote TCP stack.

@see packetSent
return False

def causedStateChange(self):
Returns true if a previous call to packetSent/packetRecvd caused a
return self._nextState != None

def getNextState(self):
Returns the class (subclass of TcpState) that represents the state that
the TCP stack should be in, as a result of previous packetSent/Recvd calls.
@return A class object, or None if no state change should occur.
return self._nextState

Interesting Predicament: Blogo does not like pasting HTML code inside of "pre" tags. It tries to make the HTML code visible, by changing each < and > to "<" or ">", respectively. It seems that I can't find a Blog client that does everything perfectly. So no Python syntax highlighting for now.


Logging Framework

I think I've got the Logging stuff set up the way I'd like it to work.

Here's an example (the doc for the responsible class)...

    >>> from loggable import Loggable
>>> Loggable().logfield('test')
2009-05-24 03:23:40,851 - loggable.Loggable - FIELD_CHANGE - test
>>> Loggable().loginfo('test')
2009-05-24 03:24:26,108 - loggable.Loggable - INFO - test
>>> class B(Loggable):
... pass
>>> B().loginfo('test')
2009-05-24 03:24:38,306 - __main__.B - INFO - test
>>> B().logpacket('packet XYZ sent')
2009-05-24 03:26:05,299 - __main__.B - PACKET_SENT - packet XYZ sent
>>> B().logresponse('response ABC generated for packet XYZ')
2009-05-24 03:26:15,745 - __main__.B - RESPONSE_GEN - response ABC generated for packet XYZ
>>> B().logfield('field abc123 set to newValue')
2009-05-24 03:26:42,292 - __main__.B - FIELD_CHANGE - field abc123 set to newValue
>>> Loggable().logstate('state changed to CONNECTED')
2009-05-24 03:31:16,848 - loggable.Loggable - STATE_CHANGE - state changed to CONNECTED


Further thoughts on design

A few more thoughts on the design of the framework:

  1. Logging (textual)
  2. Logging (packets)
  3. Diff'ing (packets)
  4. Live history

Logging (textual)
The way that the textual output of the framework is delivered needs to be well-thought-out. While it's obvious that "test passed" / "test failed" is all that should be necessary for standard runs (INFO level), there are many distinct levels below INFO that will be necessary. A few of these off the top of my head:

  • Full packet sent'd/recv'd over network IF
    This is obvious. Just a Packet sent/recv, with the BinHex print-out

  • State-change
    This should somewhat reflect the diagram provided on Page 23 of RFC 793. This behavior will also make it easier to track issues in the flow.
    1. Closed
    2. Listen
    3. SYN sent
    4. SYN recvd
    5. Established
    6. Fin Wait-1/2
    7. Close Wait
    8. Closing
    9. Time Wait
    10. Closed
  • Response generation
    Automated response generation (i.e. Send packet X, recv packet Y. Validate that Packet Y is what we were expecting, and then generate Packet Z that is a valid response/ACK for Packet Y).

  • Field generation (dynamic)
    When sending information, some fields (like IP length, checksums) may be generated dynamically/on-the-fly.

  • Manual field override
    Testing invalid states/data will also be necessary. Manually overriding fields (e.g. bad checksum) should provide extra insight. This might belong on the same logging level as Field Generation, but with a different prefix.

Logging (Packets)
Creating a TCPDUMP-formatted file with the contents of all of the packets from a test session will be invaluable in determining what, specifically, went wrong during the tests. This will make the use of third-party tools possible, and should complement the configurable logging well.

Differential Analysis
Simple packet replay, will only be useful in select circumstances. Ideally, sending packet X will always generate response Y (with some variables, such as timestamp). Playback of a TCPDUMPed session would allow this, but it would need a framework to determine what fields are important for validity, and which aren't. Since it would be difficult to require the specific lack of a particular response, this application may be very limited.

Live History
Having a history (in memory) of all of the packets sent/received during the current test will be a necessity. This is required to make sure that, for example, all of the packets were properly ack'ed as we continue sending packets. Also, it will be necessary to have internal state information about how much data has been ack'ed, and its correlation to the window size.


Saturday, May 23, 2009

Not Just OS X

The reason for my previous post, was to ensure that the issues that I encountered building the Hg-current build of PCS were not a result of running on Darwin/OSX instead of FreeBSD. This worry has been alleviated - the same issues occur on FreeBSD.


I Love X

I just wanted to note that X11, for its age, is amazing. X11 + SSH with X Forwarding + VMWare Fusion + OS X for the win. Saves me from having to run a full GUI inside the virtual machine.

(Of course, it wouldn't be necessary if FreeBSD shipped with links, lynx, or wget. I understand that there's licensing issues, and that it comes with package management and curl.)



Working on getting the latest PCS from Hg working, because it will be necessary to have proper Checksum and Length generation on-the-fly for easy generation of valid packets.

Contacted GNN with (hopefully) the last few issues to getting this to work for me.



Friday, May 22, 2009

Wireshark Behavior

Looks like the issue is the adapter.


Wireshark Strangeness

Playing around with Wireshark, looking at the dissected data, and getting some weird results. Evidently the small echo server I wrote (see below) does not create the correct TCP checksum. I thought that this might be an issue when using lo0 (loopback), but the data simply doesn't show up in Wireshark when going to/from the same IP address (in my case, Shows up in netstat:

tcp4       0      0      ESTABLISHED
tcp4 0 0 *.46002 *.* LISTEN

Not sure what its issue is.

Loopback pcap file available here.



Thinking about the best way to structure the TCP regression framework. A few things immediately jump out at me:

  • Handle either IPv4 or IPv6 gracefully
  • Reuse the most code possible
  • Easy generations of responses (e.g. create new Chain, with corresponding TCP layer with filled-in fields like Sequence number, Ack number, Ack flag, etc.)
  • Ability to test that the opposite side is actually getting a connection and recv'ing data
Handling IPv4 and IPv6 Gracefully
The simplest way to do this would be to define an interface and two implementations. However, I need to look into the differences between IPv4 and IPv6 a bit more -- static header size may preclude IPv6 from needing some tests, and there may yet be other differences.

Code Reuse
Code reuse will likely fit into the same category as above. Simple interface/implementation class hierarchy may do the trick. However, aside from the direct manipulation of the tests, there may be some tests which are a subset of other tests, or the same test may need to be run under multiple circumstances. Preventing duplicated code is something I really put a high priority on.

Response Generation
Response generation will also be a subset of the two above areas. For example, let's say I send a SYN packet, and I find the corresponding SYN/ACK packet. Having a method that generates the final ACK packet would be beneficial (or that generates an ACK packet to any packet).

Connectivity Testing
Connectivity testing should be fairly trivial. Basically, set up an echo server. If the connection is properly established, the data should be echoed back without problem. This particular solution should also lend itself to testing window sizes.

Side Note:
The blog client Blogo is amazing. Full-screen, rich-text editing FTW.


Monday, May 18, 2009

Adding functionality to PCS

Sent the below to GNN, hopefully it'll make its way into PCS.

def inet_lton(integer):
return struct.pack(">L",integer)

def inet_ltoa(integer):
return socket.inet_ntoa(inet_lton(integer))

def inet_ntol(byteString):
return struct.unpack(">L",byteString)[0]

def inet_atol(ipString):
return inet_ntol(socket.inet_aton(ipString))

Real Coding Begins!

First Step

Create a class that allows filtering of packets based on the IP address and/or port, and works with both IPv4/IPv6 and TCP/TCPv6. Since this is a TCP testing framework, this should allow a lot of isinstance(packet,pcs.packets.tcp.tcp) calls. Read more...

Thursday, May 14, 2009

Worthwhile Note

I thought it pertinent to note that I resolved the earlier-mentioned problem by setting the data member in the Packet class directly:

class Packet:
data = None

Wednesday, May 13, 2009

More PCS Issues, Bugreport

Issued a bug report on the tracker. Didn't realized that I wasn't logged in, and subsequently cannot modify the report. Oops.

Anyways, the problem is that defines the dnsheader class doesn't define the 'data' member. This is present in the Hg revision, which is why I was trying to get it working earlier.

Looks like merging won't be easy, since the newer version relies on is_tcp. Might just add a 'Data' field that sets data=None.

I am beginning to wonder why many of these variables are not defined in the Packet class body, since they are expected to exist. This would also facilitate contextual help systems (like the Python/Eclipse stuff).

Also not sure why is_tcp is used instead of isinstance(self,tcp). Looking back through the code is painful, because I haven't (to this point) found any high-level design documents. Might have to email GNN about this. Read more...

Attempted to update to PCS from Hg

Attempted to update to PCS from Mercurial. Greeted with a bunch of this:

/Users/zach/Documents/workspace/PCS/pcs/bpf/bpf.pyx:391:53: Attempting to index non-array type ''

Error converting Pyrex file to C:
li = []
n = self.bp.bf_len
if n > 0:
ip = self.bp.bf_insns
for 0 <= i < n:
li.append(op(ip[0].code, ip[0].jt, ip[0].jf, ip[0].k))

Tuesday, May 12, 2009


Finally, some time to do some investigative work with PCS.

Well, turns out that the code was doing exactly what it should have been doing, just not what I expected.

Consider the following:
sniffer = pcs.PcapConnector(options.interface)

rawPacket =
etherPacket = ethernet(rawPacket)

print "RAW"
print ByteToHex(rawPacket)
print "Ether.bytes"
print ByteToHex(etherPacket.getbytes())
print "Ether.chain().bytes"
print ByteToHex(etherPacket.chain().bytes)

This results in the output:

00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00 45 00 00 4A DD AA 40 00 40 06 54 A1 AC 10 00 0A 40 0C 1C 3C C8 5B 14 46 03 BC CB 8B 8B 23 09 EF 50 18 FF FF EE FD 00 00 2A 02 01 C4 00 1F 00 04 00 14 00 00 00 00 01 C0 00 00 00 00 00 00 00 00 00 01 08 6C 7A 61 62 61 64 61
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00 45 00 00 4A DD AA 40 00 40 06 54 A1 AC 10 00 0A 40 0C 1C 3C C8 5B 14 46 03 BC CB 8B 8B 23 09 EF 50 18 FF FF EE FD 00 00 2A 02 01 C4 00 1F 00 04 00 14 00 00 00 00 01 C0 00 00 00 00 00 00 00 00 00 01 08 6C 7A 61 62 61 64 61

Looks like the ".getBytes()" function only returns the bytes for that level (which is obviously useful with the Chain object). Read more...

Monday, May 11, 2009

Back Home

Back at home for now, moving to Lansing tomorrow afternoon. Things should really get moving :-D

Turns out that all of my Summer courses are for the first half of the summer. This may result in a disproportionate amount of time being spent after July 2, versus the time spent before it. I still plan to stick fully to all of my previously-outlined commitments, especially this project.

Course Schedule:


Saturday, May 9, 2009

Moving Back Home

I'm moving back home from the DC area on Sunday. I've got some of my stuff packed up in the car already, and depending on how tomorrow goes (packing more stuff, tying up loose ends, mailing off eBay packages, biking if there's time) I should get some more time to play with PCS.

Once I get home, I have some errands to run on Monday, and will be moving into my new place in East Lansing on Tuesday. That should give me the rest of the week to get situated, and spend some real time in PCS before the actual "coding period" begins.

Here's to hoping that GNN gets the new version pushed out soon! Looking forward to it :-D Read more...

Tuesday, May 5, 2009

Issues with PCS

Not sure why the below code doesn't print out the same values three times...

import pcs
from pcs.packets.ethernet import *
from pcs.packets.ipv4 import *
from pcs.packets.tcp import *

def main():
from optparse import OptionParser

parser = OptionParser()
parser.add_option("-i", "--interface",
dest="interface", default=None,
help="Which interface to use for testing.")
(options, args) = parser.parse_args()
sniffer = pcs.PcapConnector(options.interface)

while 1:
rawPacket =
etherPacket = ethernet(rawPacket)

if type( == ipv4:
ipv4Packet = ipv4(

if type( == tcp:
tcpPacket = tcp(

print tcpPacket
print '-------------------------------------'

Monday, May 4, 2009

Playing around with PCS

Doing some non-constructive playing around with PCS. I've decided that I may very well use Eclipse with some Python plugins for development, as I've gotten used to the IDE, along with contextual help (member, method lists, etc.) and its debugger.

I'm very, very sorry TextMate. You've served me well. Read more...

Friday, May 1, 2009

Installing PCS and Pyrex on a Mac

Found this out a while ago, but figure it might be useful to users of PCS... (since I couldn't find this information anywhere else)

  1. Download PCS from
  2. Download Pyrex from
  3. Extract Pyrex. Then run
    sudo python install

  4. Extract PCS. Then run
    sudo python install

  5. Don't bother with the MacPort of pylibpcap. Just download it from, and run
    sudo python install

  6. To make sure everything works, bring up Python and type
    import pcs
    As long as you don't get any errors, it's safe to assume everything went well :-)

In retrospect, this all is extremely straightforward. I was trying to do it the "MacPorts" way instead of just *doing* it. Would've saved myself a lot of hassle... Read more...

Some Time to Work...

Hopefully I'll have some time to get some things done this weekend, now that I don't have to worry about getting the apartment cleaned up, things sold, and paying for school.

Started getting things set up on my Mac for the project. So far I've got FreeBSD installed in a VMWare machine, Python 2.5 installed (because the MacPort of 2.6 doesn't want to, for whatever reason). Downloaded PCS and started getting it set up, although I've run into a snag getting it to find Pyrex properly. Read more...


Blog Archive

About Me