Logic Validation

Looking for some input on folks familiar with low-level TCP. I've got some general questions about proper values of Sequence # and Ack # over the course of a TCP session.

ACKing Packets

A segment on the retransmission queue is fully acknowledged if the sum of its sequence number and length is less or equal than the acknowledgment value in the incoming segment.

A packet is received with sequence number "seqn" and TCP payload with "len" octets of data. The receiving party should compare "seqn" to the variable outlined in the RFC, "rcv.nxt". If they match, "rcv.next" is incremented by "len".

The ACK number of outgoing packets is always sent to 'rcv.next'-1.

What is done with packets received out-of-order, but that are entirely valid? Buffer them, then check each one's SEQ number when rcv.nxt is incremented?

Delayed ACK response
This also raises the question of how to properly ACK a received message, when packets aren't already being set. In some instances, it may be necessary to send an empty packet with the ACK flag set. How to determine when this needs to be done?

Read more...

IP and Port Representation

Small hurdle that I tried to tackle tonight is the representation of IP addresses and ports. There are at least 4 representations of each that will need to be dealt with regularly:

- ASCII (e.g. "127.0.0.1", "80")
- Host-byte-order integer, useful for ports (16777343, 80)
- Network-byte-order integer, used in raw packets (2130706433, 20480)
- Network-byte-order byte-string, used by "inet_aton" and the like. ('\x7f\x00\x00\x01', '\x00\x00\x00P')

So, the IP address and Port were encapsulated in classes. This should make dealing with them slightly easier.

Read more...

Oh.

I was having some issues earlier wrapping my mind around why, exactly, packing strings didn't give me a string with the "\x##\x##\x##" format. Evidently, that is only necessary for characters that do not have a normal, printable equivalent.

Example:

>>> "\x7D" + "\x7B"
'}{'

Anyway, the following output is what had me confused, and why I spent about an hour or so now looking into the 'struct' package and why it wasn't giving me what I thought I wanted. Notice the plaintext "test" at the end. It seemed out of place, so I was assuming that I was doing something wrong.

>>> from pcsextension.payload import payload
>>> from pcs.packets.tcp import tcp
>>> p = payload('test')
>>> t = tcp()
>>> t.data = p
>>> t.chain().bytes
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00test'

Read more...

Rejected Email

George

If you're reading this, I'm having some issues contacting you via email:

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

gnn@neville-neil.com

Message will be retried for 2 more day(s)

Technical details of temporary failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 451 451 4.3.0 Message held for human verification before permitting delivery. For help, please quote incident ID 13393043. (state 18).

Read more...

Memorial Day

Happy Memorial Day, All!

Read more...

Logging Framework Change

Decided I was going about it all wrong (it was getting late). Changed the implementation to simply provide convenience methods on top of the Python 'logging' package's framework -- with the added benefit of not having to worry about the nitty-gritty of the 'logging' package. Just simple, direct access to the log methods.

     >>> from loggable  import Loggable
     >>> class A(object): 
     ...    def  __init__(self): 
     ...      self.log = Loggable (self) 
     ...   
     >>> a = A() 
     >>> a.log.info('test') 
     2009-05-24 12:48:08,654 - __main__.A - INFO - test 
     >>> a.log.critical('test') 
     2009-05-24 12:48:19,475 - __main__.A - CRITICAL - test 
     >>> a.log.state('test') 
     2009-05-24 12:48:22,978 - __main__.A - STATE_CHANGE - test 
     >>> a.log.pktsent ('test') 
     2009-05-24 12:48:26,402 - __main__.A - PACKET_SENT - test 
     >>> a.log.generated('test') 
     2009-05-24 12:48:49,516 - __main__.A - RESPONSE_GEN - test 
     >>> a.log.field('test') 
     2009-05-24 12:48:52,475 - __main__.A - FIELD_CHANGE - test

Also have been working on a simple TCP State Machine today. My theory is that we will need to know [1] what state the local TCP connection is in, and [2] what state the remote TCP connection should be in. A simple state machine should allow us to do this with some flexibility. Here is the base class/interface. Suggestions are highly welcomed:

class  TcpState(object):
    ''' 
    Base TCP State class.   Note that it will, by design, ignore all non-TCP layers.
    The TCP State classes will not perform verification that the packets are all
    to/from the same hosts.   That filtering will be implemented elsewhere in the
    framework.
    '''
    
    def  __init__(self):
        self._nextState = None 
    
    def  packetSent(self, tcpPacket):
        ''' 
        Hand off a packet to the TcpState object, where the object will perform
        any necessary calculation, and determine if a state change should be
        caused by the provided packet.
        
        The packet should be a packet that is considered 'sent' from
        the TCP stack that this state belongs to.
        
        @return True if the packet causes a state change.   False otherwise.
        '''
         return  False
    
    def  packetRecvd(self, tcpPacket):
        ''' 
        Same as packetSent, but in the opposite direction.
        
        The packet should be a packet that is considered 'received' from
        the remote TCP stack.
        
        @see packetSent
        '''
         return  False
    
    def  causedStateChange(self):
        ''' 
        Returns true if a previous call to packetSent/packetRecvd caused a  
        state-change.
        '''
        return  self._nextState != None 
    
    def  getNextState(self):
        ''' 
        Returns the class (subclass of TcpState) that represents the state that
        the TCP stack should be in, as a result of previous packetSent/Recvd  calls.
        @return A class object, or None if no state change should occur.
        '''
        return  self._nextState

Interesting Predicament: Blogo does not like pasting HTML code inside of "pre" tags. It tries to make the HTML code visible, by changing each < and > to "<" or ">", respectively. It seems that I can't find a Blog client that does everything perfectly. So no Python syntax highlighting for now.

Read more...

Logging Framework

I think I've got the Logging stuff set up the way I'd like it to work.

Here's an example (the doc for the responsible class)...

    >>> from loggable import Loggable
    >>> Loggable().logfield('test')
    2009-05-24 03:23:40,851 - loggable.Loggable - FIELD_CHANGE - test
    >>> Loggable().loginfo('test')
    2009-05-24 03:24:26,108 - loggable.Loggable - INFO - test
    >>> class B(Loggable):
    ...   pass
    ... 
    >>> B().loginfo('test')
    2009-05-24 03:24:38,306 - __main__.B - INFO - test 
    >>> B().logpacket('packet XYZ sent')
    2009-05-24 03:26:05,299 - __main__.B - PACKET_SENT - packet XYZ sent
    >>> B().logresponse('response ABC generated for packet XYZ')
    2009-05-24 03:26:15,745 - __main__.B - RESPONSE_GEN - response ABC generated for packet XYZ
    >>> B().logfield('field abc123 set to newValue')
    2009-05-24 03:26:42,292 - __main__.B - FIELD_CHANGE - field abc123 set to newValue
    >>> Loggable().logstate('state changed to CONNECTED')
    2009-05-24 03:31:16,848 - loggable.Loggable - STATE_CHANGE - state changed to CONNECTED

Read more...

Further thoughts on design

A few more thoughts on the design of the framework:

1. Logging (textual)
2. Logging (packets)
3. Diff'ing (packets)
4. Live history

Logging (textual)
The way that the textual output of the framework is delivered needs to be well-thought-out. While it's obvious that "test passed" / "test failed" is all that should be necessary for standard runs (INFO level), there are many distinct levels below INFO that will be necessary. A few of these off the top of my head:

• Full packet sent'd/recv'd over network IF
  This is obvious. Just a Packet sent/recv, with the BinHex print-out
  
  
• State-change
  This should somewhat reflect the diagram provided on Page 23 of RFC 793. This behavior will also make it easier to track issues in the flow.
  
  1. Closed
  2. Listen
  3. SYN sent
     
  4. SYN recvd
  5. Established
  6. Fin Wait-1/2
     
  7. Close Wait
     
  8. Closing
     
  9. Time Wait
     
  10. Closed
• Response generation
  Automated response generation (i.e. Send packet X, recv packet Y. Validate that Packet Y is what we were expecting, and then generate Packet Z that is a valid response/ACK for Packet Y).
  
  
• Field generation (dynamic)
  When sending information, some fields (like IP length, checksums) may be generated dynamically/on-the-fly.
  
  
• Manual field override
  Testing invalid states/data will also be necessary. Manually overriding fields (e.g. bad checksum) should provide extra insight. This might belong on the same logging level as Field Generation, but with a different prefix.

Logging (Packets)
Creating a TCPDUMP-formatted file with the contents of all of the packets from a test session will be invaluable in determining what, specifically, went wrong during the tests. This will make the use of third-party tools possible, and should complement the configurable logging well.

Differential Analysis
Simple packet replay, will only be useful in select circumstances. Ideally, sending packet X will always generate response Y (with some variables, such as timestamp). Playback of a TCPDUMPed session would allow this, but it would need a framework to determine what fields are important for validity, and which aren't. Since it would be difficult to require the specific lack of a particular response, this application may be very limited.

Live History
Having a history (in memory) of all of the packets sent/received during the current test will be a necessity. This is required to make sure that, for example, all of the packets were properly ack'ed as we continue sending packets. Also, it will be necessary to have internal state information about how much data has been ack'ed, and its correlation to the window size.

Read more...

Not Just OS X

The reason for my previous post, was to ensure that the issues that I encountered building the Hg-current build of PCS were not a result of running on Darwin/OSX instead of FreeBSD. This worry has been alleviated - the same issues occur on FreeBSD.

Read more...

I Love X

I just wanted to note that X11, for its age, is amazing. X11 + SSH with X Forwarding + VMWare Fusion + OS X for the win. Saves me from having to run a full GUI inside the virtual machine.

(Of course, it wouldn't be necessary if FreeBSD shipped with links, lynx, or wget. I understand that there's licensing issues, and that it comes with package management and curl.)

Read more...

PCS-Current

Working on getting the latest PCS from Hg working, because it will be necessary to have proper Checksum and Length generation on-the-fly for easy generation of valid packets.

Contacted GNN with (hopefully) the last few issues to getting this to work for me.

Zach

Read more...

Wireshark Behavior

Looks like the issue is the adapter.

http://www.mail-archive.com/wireshark-users@wireshark.org/msg03669.html

Read more...

Wireshark Strangeness

Playing around with Wireshark, looking at the dissected data, and getting some weird results. Evidently the small echo server I wrote (see below) does not create the correct TCP checksum. I thought that this might be an issue when using lo0 (loopback), but the data simply doesn't show up in Wireshark when going to/from the same IP address (in my case, 172.16.0.10). Shows up in netstat:

tcp4       0      0  172.16.0.10.46002      172.16.0.10.55074      ESTABLISHED
tcp4       0      0  172.16.0.10.55074      172.16.0.10.46002      ESTABLISHED
tcp4       0      0  *.46002                *.*                    LISTEN

Not sure what its issue is.

Loopback pcap file available here.

Read more...

Structure

Thinking about the best way to structure the TCP regression framework. A few things immediately jump out at me:

• Handle either IPv4 or IPv6 gracefully
• Reuse the most code possible
• Easy generations of responses (e.g. create new Chain, with corresponding TCP layer with filled-in fields like Sequence number, Ack number, Ack flag, etc.)
• Ability to test that the opposite side is actually getting a connection and recv'ing data
  

Handling IPv4 and IPv6 Gracefully

The simplest way to do this would be to define an interface and two implementations. However, I need to look into the differences between IPv4 and IPv6 a bit more -- static header size may preclude IPv6 from needing some tests, and there may yet be other differences.

Code Reuse

Code reuse will likely fit into the same category as above. Simple interface/implementation class hierarchy may do the trick. However, aside from the direct manipulation of the tests, there may be some tests which are a subset of other tests, or the same test may need to be run under multiple circumstances. Preventing duplicated code is something I really put a high priority on.

Response Generation

Response generation will also be a subset of the two above areas. For example, let's say I send a SYN packet, and I find the corresponding SYN/ACK packet. Having a method that generates the final ACK packet would be beneficial (or that generates an ACK packet to any packet).

Connectivity Testing

Connectivity testing should be fairly trivial. Basically, set up an echo server. If the connection is properly established, the data should be echoed back without problem. This particular solution should also lend itself to testing window sizes.

Side Note:
The blog client Blogo is amazing. Full-screen, rich-text editing FTW.

Read more...

Adding functionality to PCS

Sent the below to GNN, hopefully it'll make its way into PCS.

def inet_lton(integer):
    return struct.pack(">L",integer)

def inet_ltoa(integer):
    return socket.inet_ntoa(inet_lton(integer))
    
def inet_ntol(byteString):
    return struct.unpack(">L",byteString)[0]

def inet_atol(ipString):
    return inet_ntol(socket.inet_aton(ipString))

Real Coding Begins!

First Step

Create a class that allows filtering of packets based on the IP address and/or port, and works with both IPv4/IPv6 and TCP/TCPv6. Since this is a TCP testing framework, this should allow a lot of isinstance(packet,pcs.packets.tcp.tcp) calls. Read more...

Worthwhile Note

I thought it pertinent to note that I resolved the earlier-mentioned problem by setting the data member in the Packet class directly:

class Packet:
  ...
  data = None
  ...

Read more...

More PCS Issues, Bugreport

Issued a bug report on the SF.net tracker. Didn't realized that I wasn't logged in, and subsequently cannot modify the report. Oops.

Anyways, the problem is that dns.py defines the dnsheader class doesn't define the 'data' member. This is present in the Hg revision, which is why I was trying to get it working earlier.

Looks like merging won't be easy, since the newer version relies on is_tcp. Might just add a 'Data' field that sets data=None.

I am beginning to wonder why many of these variables are not defined in the Packet class body, since they are expected to exist. This would also facilitate contextual help systems (like the Python/Eclipse stuff).

Also not sure why is_tcp is used instead of isinstance(self,tcp). Looking back through the code is painful, because I haven't (to this point) found any high-level design documents. Might have to email GNN about this. Read more...

Attempted to update to PCS from Hg

Attempted to update to PCS from Mercurial. Greeted with a bunch of this:

/Users/zach/Documents/workspace/PCS/pcs/bpf/bpf.pyx:391:53: Attempting to index non-array type ''

Error converting Pyrex file to C:
------------------------------------------------------------
...
        li = []
        n = self.bp.bf_len
        if n > 0:
            ip = self.bp.bf_insns
            for 0 <= i < n:
                li.append(op(ip[0].code, ip[0].jt, ip[0].jf, ip[0].k))
                                                              ^
------------------------------------------------------------

Read more...

Finally!

Finally, some time to do some investigative work with PCS.

Well, turns out that the code was doing exactly what it should have been doing, just not what I expected.

Consider the following:

sniffer = pcs.PcapConnector(options.interface)

rawPacket = sniffer.read()
etherPacket = ethernet(rawPacket)
     
print "RAW"
print ByteToHex(rawPacket)
print "Ether.bytes"
print ByteToHex(etherPacket.getbytes())
print "Ether.chain().bytes"
print ByteToHex(etherPacket.chain().bytes)

This results in the output:

RAW
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00 45 00 00 4A DD AA 40 00 40 06 54 A1 AC 10 00 0A 40 0C 1C 3C C8 5B 14 46 03 BC CB 8B 8B 23 09 EF 50 18 FF FF EE FD 00 00 2A 02 01 C4 00 1F 00 04 00 14 00 00 00 00 01 C0 00 00 00 00 00 00 00 00 00 01 08 6C 7A 61 62 61 64 61
Ether.bytes
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00
Ether.chain().bytes
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00 45 00 00 4A DD AA 40 00 40 06 54 A1 AC 10 00 0A 40 0C 1C 3C C8 5B 14 46 03 BC CB 8B 8B 23 09 EF 50 18 FF FF EE FD 00 00 2A 02 01 C4 00 1F 00 04 00 14 00 00 00 00 01 C0 00 00 00 00 00 00 00 00 00 01 08 6C 7A 61 62 61 64 61

Looks like the ".getBytes()" function only returns the bytes for that level (which is obviously useful with the Chain object). Read more...

Back Home

Back at home for now, moving to Lansing tomorrow afternoon. Things should really get moving :-D

Turns out that all of my Summer courses are for the first half of the summer. This may result in a disproportionate amount of time being spent after July 2, versus the time spent before it. I still plan to stick fully to all of my previously-outlined commitments, especially this project.

Course Schedule:
HTML
iCal

Embedded:
Read more...

Moving Back Home

I'm moving back home from the DC area on Sunday. I've got some of my stuff packed up in the car already, and depending on how tomorrow goes (packing more stuff, tying up loose ends, mailing off eBay packages, biking if there's time) I should get some more time to play with PCS.

Once I get home, I have some errands to run on Monday, and will be moving into my new place in East Lansing on Tuesday. That should give me the rest of the week to get situated, and spend some real time in PCS before the actual "coding period" begins.

Here's to hoping that GNN gets the new version pushed out soon! Looking forward to it :-D Read more...

Issues with PCS

Not sure why the below code doesn't print out the same values three times...

import pcs
from pcs.packets.ethernet import *
from pcs.packets.ipv4 import *
from pcs.packets.tcp import *

def main():
    from optparse import OptionParser

    parser = OptionParser()
    parser.add_option("-i", "--interface",
                      dest="interface", default=None,
                      help="Which interface to use for testing.")
    (options, args) = parser.parse_args()
    sniffer = pcs.PcapConnector(options.interface)
    
    while 1:
        rawPacket = sniffer.read()
        etherPacket = ethernet(rawPacket)
        
        if type(etherPacket.data) == ipv4:
            ipv4Packet = ipv4(etherPacket.data.getbytes())
            
            if type(ipv4Packet.data) == tcp:
                tcpPacket = tcp(ipv4Packet.data.getbytes())
                
                print etherPacket.data.data
                print ipv4Packet.data
                print tcpPacket
        print '-------------------------------------'

Read more...

Playing around with PCS

Doing some non-constructive playing around with PCS. I've decided that I may very well use Eclipse with some Python plugins for development, as I've gotten used to the IDE, along with contextual help (member, method lists, etc.) and its debugger.

I'm very, very sorry TextMate. You've served me well. Read more...

Installing PCS and Pyrex on a Mac

Found this out a while ago, but figure it might be useful to users of PCS... (since I couldn't find this information anywhere else)

1. Download PCS from SF.net
   
2. Download Pyrex from canterbury.ac.nz
   
3. Extract Pyrex. Then run

   sudo python setup.py install

   
   
4. Extract PCS. Then run

   sudo python setup.py install

   
   
5. Don't bother with the MacPort of pylibpcap. Just download it from SF.net, and run

   sudo python setup.py install

   
   
6. To make sure everything works, bring up Python and type

   import pcs

   As long as you don't get any errors, it's safe to assume everything went well :-)
   

In retrospect, this all is extremely straightforward. I was trying to do it the "MacPorts" way instead of just *doing* it. Would've saved myself a lot of hassle... Read more...

Some Time to Work...

Hopefully I'll have some time to get some things done this weekend, now that I don't have to worry about getting the apartment cleaned up, things sold, and paying for school.

Started getting things set up on my Mac for the project. So far I've got FreeBSD installed in a VMWare machine, Python 2.5 installed (because the MacPort of 2.6 doesn't want to, for whatever reason). Downloaded PCS and started getting it set up, although I've run into a snag getting it to find Pyrex properly. Read more...