gsoc-tcpregression

Holy Crap How Time Flies

2009-11-16T21:22:00.002-05:00

Lots of time has gone by since I've had a chance to put some hours into the TCP Regression framework. I haven't forgotten it, just been busy. A trip to New York City, followed by Halloween, and two absolutely gorgeous November sundays kept me away from the computer for the past few weeks.

The coming weekends, since I'm not going home for Thanksgiving, as well as Christmas-time (since most of the other interns leave Dec. 17, and I am staying until Dec. 31) will probably see a spark of productivity.

Build Issues when Installing

2009-10-11T14:33:00.002-04:00

In the process of investigating the build issues on Mac OS X 10.6 Snow Leopard, I decided to build on FreeBSD.

The combo:

python setup.py config

python setup.py build

sudo python setup.py test

Works just fine. However, once the package is installed... things go haywire (full output below). The issue appears to be in the final linking stage.

make

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

make install

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

For whatever reason, libpcap isn't being linked in. The tests attempt to import pcap, then make sure that it is valid, and if that doesn't work, import pcs.pcap (as seen at the end). When attempting to import the installed version of pcap, the "import pcap" statement fails. Grrrr...

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ ls

CHANGES LICENSE Makefile README pcap.c pcap.pyx pcap_ex.c pcap_ex.h setup.py tests testsniff.py

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ make

pyrexc pcap.pyx

python setup.py config

running config

found {'libraries': ['pcap'], 'library_dirs': ['/usr/lib'], 'include_dirs': ['/usr/include']}

python setup.py build

running build

running build_ext

pyrexc pcap.pyx --> pcap.c

building 'pcap' extension

creating build

creating build/temp.freebsd-7.1-RELEASE-i386-2.5

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o

pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':

pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':

pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_dump':

pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o

pcap_ex.c: In function 'pcap_ex_next':

pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type

creating build/lib.freebsd-7.1-RELEASE-i386-2.5

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test

python setup.py test

running test

testDispatch (tests.testPcap.TestPcap) ... ok

testEnumerateInterfaces (tests.testPcap.TestPcap) ... ok

testErrors (tests.testPcap.TestPcap) ... ok

testInjectedPacketIsReceived (tests.testPcap.TestPcap) ... ok

testIter (tests.testPcap.TestPcap) ... ok

testIterable (tests.testPcap.TestPcap) ... ok

testOpenDefaultInterface (tests.testPcap.TestPcap) ... ok

testOpenLive (tests.testPcap.TestPcap) ... ok

testPacketFilter (tests.testPcap.TestPcap) ... ok

testProperties (tests.testPcap.TestPcap) ... ok

testReadpkts (tests.testPcap.TestPcap) ... ok

testWriteDump_ReadOffline (tests.testPcap.TestPcap) ... ok

----------------------------------------------------------------------

Ran 12 tests in 1.541s

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make install

python setup.py install

running install

running build

running build_ext

pyrexc pcap.pyx --> pcap.c

building 'pcap' extension

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o

pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':

pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':

pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_dump':

pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o

pcap_ex.c: In function 'pcap_ex_next':

pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

running install_lib

copying build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so -> /usr/local/lib/python2.5/site-packages

running install_egg_info

Removing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info

Writing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test

python setup.py test

running test

Traceback (most recent call last):

File "setup.py", line 120, in

provides = ['pcap'] )

File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup

dist.run_commands()

File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands

self.run_command(cmd)

File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command

cmd_obj.run()

File "setup.py", line 91, in run

from tests.testPcap import PcapTestSuite

File "/usr/home/zjriggl/zachriggle-pypcap-1f0484c/tests/testPcap.py", line 17, in

import pcs.pcap as pcap

ImportError: No module named pcs.pcap

*** Error code 1

Stop in /usr/home/zjriggl/zachriggle-pypcap-1f0484c.

PCS-0.5 Diff with PCS version used in TcpRegression

2009-10-11T12:35:00.003-04:00

I've created a Diff file that shows all of the changes that I made to PCS over the course of the summer. Some of these might be merged back into the PCS mainline, others might not.

The highlight of the changes are:

BitString class. The code for encoding individual bits, even if there is a lack of alignment on 8-bit boundaries, is a bit iffy (even the code says "This makes my head hurt"). The BitString class allows manipulation of individual bits just like a Python list object, and can be translated into an integer value, string value, binary string (via bin()), hex string (via hex()), and allows arbitrary manipulation.
The names of the pcs.packets.tcp.tcp class fields were assigned variables. Instead of object['fieldName'] which is prone to typos going undetected, I made a bunch of variables so that they can be referred to as object[f_fieldname]. If mistyped, the interpreter picks up on the nonexistent variable.

GNN: I've explicitly emailed you about this, as this was something that you requested.

Everyone else can get a copy here:

http://p4db.freebsd.org/fileDownLoad.cgi?FSPC=//depot/projects/soc2009/zjriggl%5ftcpregression/diff/pcs%2d0.5.diff&REV=1

From looking at the diff, it does not appear that my changes rely on anything in the forked PCap library (and for that matter, don't involve PCap at all).

Zach

Yep, 10.6 Did It (Methinks)

2009-09-20T18:25:00.003-04:00

The build.py script was only set to look in the first include directory for pcap.h. On 10.6, /usr/include/pcap.h includes /usr/local/pcap/pcap.h, which contains all of the definitions the build system was looking for.

Fixed the build script, will add it to Git soon.

10.6 Breaks PyPcap?

2009-09-13T19:42:00.000-04:00

I think Snow Leopard breaks PyPcap -- for whatever reason, I'm getting compile errors that didn't previously exist. I'll have to boot up one of the VMs and see if everything still works OK, or if there was some change that I didn't properly test that I also completely forgot that I made in the first place.

Docs Added

2009-08-29T11:49:00.001-04:00

Lots of docs added to some of the core components. I'm on my way to the Apple store to have them fix this piece-o-junk, so hopefully I'll have it back soon!

Committed all changes

2009-08-17T03:04:00.001-04:00

All changes have been committed to P4 (TcpRegression) and Git (PyPcap). Eveyrthing works on my dev box, but we all know that means nothing. I'll sort out any odds and ends tomorrow. I need to get up for work in 3 hours.

- Zach

And so it ends

2009-08-16T23:28:00.001-04:00

SoC is coming to an end tonight.

Overall, I think the project went very well. I'm going to try to bust out a few deliverable tests tonight since those were the original goal, but I think the volume of work that I got done will speak for itself.

I forked PyPcap, fixed bugs, extended the functionality, wrote better tests.
I forked PCS-0.5, fixed bugs, extended the functionality (although I didn't write tests for PCS).
I've essentially implemented a reference userland implementation of TCP on top of PCS and PyPcap, which should allow for even more applications than a simple regression testing suite, complete with tests of the code itself, as well as a few deliverable tests that should validate functionality of various TCP stacks.

In addition to continuing work with the TcpRegression suite, I've sent an email to the Metasploit project to see if they could use another leisure-pace developer. That should be an interesting project, and maybe I'll be able to apply some of the knowledge about TCP that I gained over the summer. Overall, it's been very productive - for my personal benefit (fiscal and intellectual), and hopefully the FOSS community and FreeBSD as well.

I've got a few minor changes to PyPcap that need to go up -- mostly it's just the inclusion of a function that will print out a string of bytes in the same format that they appear in Wireshark's Packet Bytes view.

There's a load of TcpRegression functionality and tests that will be in the next commit. Over the next week or so, I'll work on the documentation and cleaning up the code, and getting Google their code sample. I'd like a nice, solid "0.1" release. It's also crossed my mind to separate the regression tests themselves from the main framework, and re-badging it "TCPython". The name looks like it hasn't been taken yet, but we'll see how that goes.

Special thanks to Titus Brown and George Neville-Neil for helping me throughout the summer, you both helped me out of a few ruts along the way that could've made the whole project a lot less enjoyable.

Published changes...

2009-08-16T06:12:00.001-04:00

Published changes to the PyPcap library. Get the new hotness here:

http://zachriggle.github.com/pypcap/
or
git clone git://github.com/zachriggle/pypcap

This build tested on Mac OS X 10.5.7 and FreeBSD 7.1

CHANGELOG

pypcap-1.2:
- included changes from George Neville-Neil's PCS (http://pcs.sf.net)
- added more tests
- added logging functionality
- implemented patches provided by several users from the original PyPcap site on Google Code. Thanks to...
getxsick
dirk-loss.de

- Changed some interfaces, but made sure to allow backwards compatibility.
Examples:

__init__ now has separate args for interface, filename, and dump file.
dump_close from PCS is now closeDumps
the filter can be set by accessing the '.filter' property
each pcap object can now be re-opened after closing with

- openLive
- openOffline
- openDump

- exposed findalldevs() [getxsick]
- exposed lookupnet() [getxsick]
- exposed 'cnt' arg of loop() [getxsick]
- fixed setnonblock, loop() [dirk-loss.de]

Busy Day

2009-08-16T05:04:00.001-04:00

Busy day. Added some more PyPcap functionality, fixed some bugs, and implemented tests. All of the functionality is at least touched by the tests...
- Live capture
- Dumping to file
- Reading from file
- Packet filter

Additionally, migrated the 'tcpfilter' class to use bpf instead of manually inspecting the fields of each packet after it had been constructed by PCS. This should be MUCH faster.

Also learned that in Python...

class X():
....y = someObject()

... means that all X objects will be instantiated with the *same instance* of someObject, that is instantiated when the class is declared, instead of each object being allocated when the class instance is created. Weird. I thought it was run when the class was instantiated, and just allowed for the constructor to be a little bit less cluttered. Guess I was wrong (or that there's a Python bug).

C'mon Nose, stop being a stupid WHAAAAAAAAAT

2009-08-13T22:17:00.001-04:00

In python...

>>> t = TestPcap('')
>>> t.testEnumerateInterfaces()
>>> t.testOpenDefaultInterface()
>>> t.testOpenSpecificInterface()

But...

zach@Zachs-Computer:~/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests$nosetests testPcap.py
E.E
======================================================================
ERROR: testEnumerateInterfaces (testPcap.TestPcap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 33, in testEnumerateInterfaces
listOfIfs = pcap.findalldevs()
AttributeError: 'module' object has no attribute 'findalldevs'

======================================================================
ERROR: testOpenSpecificInterface (testPcap.TestPcap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 38, in testOpenSpecificInterface
iface = pcap.findalldevs()[0]
AttributeError: 'module' object has no attribute 'findalldevs'

----------------------------------------------------------------------
Ran 3 tests in 0.004s

FAILED (errors=2)

Here's the output of a "print dir(pcap)" from within the test, run through nosetests.

-------------------- >> begin captured stdout << ---------------------
['DLT_ARCNET', 'DLT_AX25', 'DLT_CHAOS', 'DLT_EN10MB', 'DLT_EN3MB', 'DLT_FDDI', 'DLT_IEEE802', 'DLT_LINUX_SLL', 'DLT_LOOP', 'DLT_NULL', 'DLT_PFLOG', 'DLT_PFSYNC', 'DLT_PPP', 'DLT_PRONET', 'DLT_RAW', 'DLT_SLIP', '__author__', '__builtins__', '__copyright__', '__doc__', '__file__', '__license__', '__maintainer__', '__name__', '__revison__', '__url__', '__version__', 'bpf', 'calendar', 'dltoff', 'ex_name', 'lookupdev', 'pcap', 'sys', 'time']
--------------------- >> end captured stdout << ----------------------

Obviously, it's not there. Pcap is imported as:

try:
import pcs.pcap as pcap
except:
import pcs

Here's the output of "print pcs.__file__" from Python:
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/pcap.so

Aaaand from Nose-tests:
/Library/Python/2.5/site-packages/pcs/pcap.so

Whaaaaaaaaaaat?

Time

2009-08-13T01:25:00.001-04:00

Lost track of time tonight, stayed up way later than I wanted to. Overhauled a LOT of code in the PyPcap library (both C any Python code). It's now a lot cleaner, and a lot of the functionality that was only available via creating a new pcap object is now available at runtime (i.e. open/reopen an interface or offline capture).

Also changed a bit of the Windows code that (for whatever reason) attempted to implement pcap_lookupdev, even though WinPcap has that function. Other minor changes were included in the C code (a little bit of refactoring).

The Python code now includes logging for most of its functionality, which is disabled by default. It does require either [1] modifying the PYX file or [2] modifying pcap.DEBUG_LEVEL before instantiating the pcap object that logging is desired for. Lots of refactoring. Hopefully I didn't break anything.

Must. Get. To. Sleeeeeeeeeeep.

Nope, it was Python.

2009-08-12T21:31:00.001-04:00

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html

The __next__ method

Extension types wishing to implement the iterator interface should define a method called __next__, not next. The Python system will automatically supply a next method which calls your__next__. Do NOT explicitly give your type a next method, or bad things could happen.

Wtf?

2009-08-12T21:29:00.001-04:00

Changing the name of "__next__"... what the hell, Python? (Although I'm much more apt to blame Pyrex)

File "<stdin>", line 1, in <module>

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read
return self.pcapHandle.readpkt()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt
packet = self.read()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read
packet = self.file.next()

Yep.

2009-08-12T21:12:00.001-04:00

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read
return self.pcapHandle.readpkt()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt
packet = self.read()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read
packet = self.file.next()
File "pcap.pyx", line 502, in pcap.pcap.__next__
KeyboardInterrupt

Not understanding the logic flow here

2009-08-12T20:59:00.001-04:00

Going back and testing stuff. Check this out...

self.filter.read() # tcpregression.tcpfilter.TcpFilter.read()

which calls:

return self.pcapHandle.readpkt() # pcs.PcapConnector.readpkt()

which calls:

packet = self.file.next()[1] # pcs.pcap.pcap.next()

However, debug statements show

"IN __NEXT__".

Dubbleyou Tee Eff.

HA!

2009-08-12T18:11:00.003-04:00

Finally figured out the logging bit... a few changes to the config file, combined with:

logFiles  = [ 'logging.conf',
             join( expanduser( '~'  ), 'logging.conf'  ), 
             join( dirname( __file__ ), 'logging.conf'  ), 
             None ] 
  
for  logFile in  logFiles: 
     if  exists( logFile ): 
         print  "Using  logfile  %s"  % logFile 
         logging.config.fileConfig( logFile )

Fixed the problem

Modem dead

2009-08-10T22:34:00.001-04:00

Posting from iPhone. My router & modem fried in storm. :-/ fml

The Issue [Nevermind, Fixed]

2009-08-10T21:11:00.001-04:00

Nevermind, in the process of writing this post, I made a change at some point that made everything work all honky-dory.

build.py in pcs-0.5 and pypcap both use the following command to link everything:

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap_ex.o -L/usr/lib -lpcap -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so

build.py in tcpregression links it with...

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap_ex.o -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so

The discrepancy between the two being:

-L/usr/lib -lpcap

Exactly why this is the case, I'm unsure. Obviously, if pcap.so isn't linked with libpcap (the actual pcap library), there would be issues.

The build system uses setup.py from pcs-0.5, with a modification to include the different source directory. Here's the diff between pcs' setup.py and my "pcssetup.py". All of the changes have to do with the source being in a "src" folder.

< pcap_cache = 'src/pcs/pcap/config.pkl'
---
> pcap_cache = 'pcs/pcap/config.pkl'
71c71
<         f = open( 'src/pcs/pcap/config.h', 'w' )
---
>         f = open( 'pcs/pcap/config.h', 'w' )
122c122
<                  sources = [ 'src/pcs/pcap/pcap.pyx', 'src/pcs/pcap/pcap_ex.c' ],
---
>                  sources = [ 'pcs/pcap/pcap.pyx', 'pcs/pcap/pcap_ex.c' ],
129a130,140
> 
> setup( name = 'pcs',
>       version = '0.5',
>       description = 'Packet Construction Set',
>       author = 'George V. Neville-Neil',
>       author_email = 'gnn@neville-neil.com',
>       url = 'http://pcs.sf.net',
>       packages = ['pcs', 'pcs.packets'],
>       cmdclass = pcap_cmds,
>       ext_modules = [ pcap ],
>       )

And then the actual setup file....

from distutils.core import setup, Extension
import pcssetup
setup( name = 'tcpregression',
version = '1.0',
description = 'FreeBSD TCP Regression Suite',
author = 'Zach Riggle',
author_email = 'zjriggl@freebsd.org',
url = 'http://freebsd.org',
packages = ['tcpregression',
'tcpregression.pcsextension',
'tcpregression.tests',
'pcs',
'pcs.packets'],
package_dir = {'':'src'},
cmdclass = pcssetup.pcap_cmds,
ext_modules = [pcssetup.pcap],
)

As you can see, not too many changes.

Proper Building with Setup.py

2009-08-10T18:07:00.001-04:00

Thought I'd give proper building with "setup.py" a go. Back to square one with:

ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/build/lib.macosx-10.5-i386-2.6/pcs/pcap.so
Expected in: dynamic lookup

Humbug.

Massive Cleanup

2009-08-10T00:03:00.001-04:00

I'm going back through a lot of the code, removing commented-out code, deleting un-used files, and restructuring a bit. Hopefully this'll make everything a lot cleaner for when I'm done :-)

Problem Solved

2009-08-09T22:32:00.001-04:00

Evidently "setup.py build_ext -i" doesn't do somethign that the regular build operation does. Not sure what. Anyway, the Python documentation for setup.py says that the output files will always end up in build/lib/ or build/lib.arch/ so a "cp build/lib*/pcap.so ..." will take care of it. Hopefully that'll solve the problem.

PyPcap issues when running in-directory

2009-08-09T21:32:00.001-04:00

Some of the issues I originally had when getting PyPcap setup back in June are re-surfacing, now that I've got PCS and PyPcap (as pcs.pcap) inside of the tcpregression folder:

>>> import pcs.pcap
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pcs/__init__.py", line 68, in <module>
import pcs.pcap as pcap
ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/src/tcpregression/pcs/pcap.so
Expected in: dynamic lookup

Build system

2009-08-08T20:37:00.001-04:00

I've integrated the PyPcap changes (http://zachriggle.github.com/pypcap/) into the TcpRegression Perforce tree, and put the custom-modified PCS-0.5 directory in there, as well. It contains some bugfixes for PCS-0.5, as well as a few modifications to make things easier for my particular uses.

The build system for building PyPcap, and putting the module into the PCS directory is also in place. That was kind of an interesting foray into make and setup.py, but it works very cleanly now that I've got it figured out (thanks Titus!).

Next steps are to write a few quick tests for the TcpRegression library itself, mainly to make sure that the threading stuff works as expected. Assuming that it does, I'll be able to crank out some TCP tests and have a final product for FreeBSD.

My work schedule changes quite a bit next week, so I'll be awake while most people are asleep/at work (depending on how I do it). Basically I'm working 0200-1000 next week, and need to figure out if I'm gonna wake just before 2AM, or go to bed just after 10AM.

GitHub site, setup.py issues

2009-08-06T21:16:00.001-04:00

"git push".

Now the stuff is ACTUALLY on the site. My god is the default color scheme ugly. ~~Working to fix that!~~

http://zachriggle.github.com/pypcap

If anybody is familiar with setup.py, please contact me via email.

PyPcap Migrated to Git

2009-08-05T22:52:00.001-04:00

http://github.com/zachriggle/pypcap

A bunch of changes have been made. I'm working on going back in and documenting those changes. I'll have a changelog shortly, as well as a demo.

Forked, Fixed, Updating

2009-08-05T21:28:00.001-04:00

Taking some advice and forking PyPcap. Working on adding the patches provided by users. Right now I'm trying to get Pyrex to like the changes that I'm making to make the 'findalldevs' function closer to the real one. A user submitted a patch on the Google Code site, but it doesn't return ALL of the data the function should. Unfortunately, I have to add a bunch of types and unroll several linked-lists and arrays to get all of the data.

Perforce Woes

2009-08-02T22:46:00.001-04:00

I don't know if I mentioned this earlier, but Perforce is *by far* the hardest version management system that I've ever had to work with -- out of CVS and SVN, that is.

First off, it defines its operation off of environment variables. This makes manipulation pretty hard unless I define environment variables in bashrc or something -- and changing environments pretty hard to boot.
Second off, it complains every time my hostname changes. Right now, it keeps complaining that I can only commit from "Zachs-Computer", which is the hostname.
Third, I have to define a workspace. This in itself is a pain. I can see the benefits, although if I have to change my hostname to use a workspace, it's kind of moot point. "svn co path/to/server" FTW.

I'm back off to fight with P4 while I try to commit my code :-\

PyPcap = Abandonware

2009-08-02T19:43:00.001-04:00

So PyPcap, the Pcap library that PCS includes in its distribution, is essentially abandonware. Hasn't been a new release since Jan 2007. The files in the included PyPcap say that GNN is the maintainer.

George, if you are reading this, please let me know so that I may submit patches to PyPcap. There is at least one issue that is of high importance to my project, for which a patch already exists, that I don't mind integrating into the PyPcap project on Google Code.

I tweeted at @dugsong, so hopefully the guy in charge of the project (as far as Google Code is concerned) will give me access to make the changes. I can make the changes locally, but there's no reason not to include them online for all to use.

Okay, so that's done... but we're stuck with a hanging thread [HALP!]

2009-08-01T19:45:00.001-04:00

The full TCP exchange of 3-way-handshake, data transfer (with acknowledgement) and active-close (Established -> Fin Wait 1 -> Fin Wait 2 -> Time Wait) has been successfully tested, and works fine.

Here's the script, and subsequent output:

import testconfig
from tcpstatemachine import *
from threading import Thread
from testserver import runServer
from multiprocessing import Process

def test():
    t = TcpStateMachine()
    t.die = False
    
    serverProcess = Process(target=runServer)
    serverProcess.start()
    
    t.open()
    t.send('COOOOOOOL!')
    t.close()
    
    serverProcess.terminate()

>>> import test
2009-08-01 17:35:18,212 - tcpFilter - INFO - Opened lo0
>>> test.test()
2009-08-01 17:35:19,867 - TcpStateMachine - DEBUG - __init__ - (('127.0.0.3', 31765), ('127.0.0.1', 32261))
2009-08-01 17:35:19,870 - tcpFilter - DEBUG - openInterface - Tried to re-open same interface: lo0
2009-08-01 17:35:19,870 - tcpFilter - INFO - Opened lo0
2009-08-01 17:35:19,870 - TcpStateMachine - INFO - Local host: 127.0.0.3:31765
2009-08-01 17:35:19,871 - TcpStateMachine - INFO - Remote host: 127.0.0.1:32261
~~~~~ In runServer
~~~~~ Listening on 127.0.0.1:32261
~~~~~ Waiting for accepted connection
2009-08-01 17:35:19,950 - TcpStateMachine - DEBUG - open - In valid state to open connection
2009-08-01 17:35:19,967 - TcpStateMachine - DEBUG - open - Generated ISS: 1036344287
2009-08-01 17:35:19,969 - TcpStateMachine - DEBUG - open - Sending SYN packet
2009-08-01 17:35:19,975 - TcpStateMachine - STATE - Setting state from Closed to Syn-Sent
2009-08-01 17:35:19,976 - TcpRecvDaemon - INFO - Starting receive thread for object TcpStateMachine( (127.0.0.3,31765), (127.0.0.1,32261) )
2009-08-01 17:35:19,977 - TcpSendDaemon - INFO - Starting send thread for object TcpStateMachine( (127.0.0.3,31765), (127.0.0.1,32261) )
2009-08-01 17:35:19,980 - TcpStateMachine - DEBUG - open - Waiting for state == ESTABLISHED... Outbound Sequenes: segmentBuffer(limit=4294967296,base=1036344287,copyList=['[SYN]']+[])
2009-08-01 17:35:20,047 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344287 to 1036345633 in ['[SYN]']
2009-08-01 17:35:20,049 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344287 to 1036344288: ['[SYN]']
2009-08-01 17:35:20,050 - TcpSendDaemon - DEBUG - sendThread - Haz a SYN!
2009-08-01 17:35:20,054 - TcpStateMachine - SENT - [<class tcp Seq: 1036344287, 31765->32261, [S], reserved: 0, checksum: 7711, offset: 5, window: 65535L>, 
<class payload {payload: ''}>]
2009-08-01 17:35:20,484 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:20,485 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201111, 32261->31765, [AS], reserved: 0, checksum: 65058, offset: 6, window: 65535, ack_number: 1036344288>]
2009-08-01 17:35:20,487 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] IN SYN_SENT
2009-08-01 17:35:20,489 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Has ACK
2009-08-01 17:35:20,490 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Accepted ACK
2009-08-01 17:35:20,492 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Has SYN and either good ACK or no ACK
2009-08-01 17:35:20,492 - TcpStateMachine - STATE - Setting state from Syn-Sent to Established
Sending ack: {seq: 1036344288, ack: 1, ack#: 1883201112}
2009-08-01 17:35:20,497 - TcpStateMachine - SENT - [<class tcp Seq: 1036344288, 31765->32261, [A], reserved: 0, checksum: 20344, offset: 5, window: 65535L, ack_number: 1883201112L>]
~~~~~ Connected
~~~~~ Waiting on Data
2009-08-01 17:35:20,499 - TcpStateMachine - DEBUG - open - Established
2009-08-01 17:35:20,507 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344288 to 1036345634 in ['C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!', 'C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!']
2009-08-01 17:35:20,508 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344288 to 1036344308: ['C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!', 'C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!']
2009-08-01 17:35:20,511 - TcpStateMachine - SENT - [<class tcp Seq: 1036344288, 31765->32261, [A], reserved: 0, checksum: 21671, offset: 5, window: 65535L, ack_number: 1883201112L>, 
<class payload {payload: 'COOOOOOOL!COOOOOOOL!'}>]
Received Data: COOOOOOOL!COOOOOOOL!
~~~~~ Waiting on Data
2009-08-01 17:35:21,008 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:21,009 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [A], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344308>]
2009-08-01 17:35:21,009 - TcpStateMachine - ERROR - This point should never be reached!
2009-08-01 17:35:21,059 - TcpStateMachine - STATE - Setting state from Established to Fin-Wait-1
2009-08-01 17:35:22,021 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344308 to 1036345654 in ['[FIN]']
2009-08-01 17:35:22,023 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344308 to 1036344309: ['[FIN]']
2009-08-01 17:35:22,024 - TcpSendDaemon - DEBUG - sendThread - Haz an FIN!
2009-08-01 17:35:22,028 - TcpStateMachine - SENT - [<class tcp Seq: 1036344308, 31765->32261, [AF], reserved: 0, checksum: 20323, offset: 5, window: 65535L, ack_number: 1883201112L>, 
<class payload {payload: ''}>]
Received Data: 
~~~~~ No more data
2009-08-01 17:35:23,016 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:23,017 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [A], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344309>]
2009-08-01 17:35:23,019 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-1] Our FIN is ACKed
2009-08-01 17:35:23,019 - TcpStateMachine - STATE - Setting state from Fin-Wait-1 to Fin-Wait-2
2009-08-01 17:35:23,020 - TcpStateMachine - ERROR - This point should never be reached!
2009-08-01 17:35:23,023 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:23,024 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [AF], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344309>]
2009-08-01 17:35:23,025 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-2] connection closing
2009-08-01 17:35:23,027 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-2] sending ACK of FIN
Sending ack: {seq: 1036344309, ack: 1, ack#: 1883201113}
2009-08-01 17:35:23,029 - TcpStateMachine - SENT - [<class tcp Seq: 1036344309, 31765->32261, [A], reserved: 0, checksum: 20322, offset: 5, window: 65535L, ack_number: 1883201113L>]
2009-08-01 17:35:23,030 - TcpStateMachine - STATE - Setting state from Fin-Wait-2 to Time-Wait
2009-08-01 17:35:23,031 - TcpStateMachine - DEBUG - debugWithState - [Time-Wait] Starting TIME_WAIT timer
2009-08-01 17:35:24,035 - TcpSendDaemon - DEBUG - sendThread - Quitting send thread
2009-08-01 17:35:33,032 - TcpStateMachine - STATE - Setting state from Time-Wait to Closed

It's a bit more verbose than it needs to be, and there's a few things that will be taken out (i.e. "Haz a SYN!") but overall I'm pleased with the way it works.

However, the only remaining problem before the real-real test-writing gets going is that the call to retrieve the next packet from PCAP is a blocking call that's performed by a Python thread (which has no terminate() or equivalent method).

Looking into the issue a bit, pypcap has a "setnonblock(nonblock=True)" method, but due to the way everything is done, this isn't quite so easy to work with (there are also some bug reports that say that this functionality doesn't necessarily work as expected).

Diagram

2009-07-31T00:18:00.001-04:00

Sexy.

http://www.eventhelix.com/RealtimeMantra/Networking/Tcp.pdf

BitString Cleanup

2009-07-24T11:49:00.001-04:00

Finally settled on the below design for BitString. I had added, and then removed, functionality for AND, OR, and XOR because it is ambiguous as to what length to use if the operands are not the same length.

For example, it should be possible to AND
0xFFFF & 0x7
To get 0x7.

However, is that '0b0000000000000111' or '0b0111' or some other variant. Given that the values are indexes from the most-significant-bit, this creates an issue.

Anyway, here it is below. Modding PCS to use this for decoding simple values.

class  BitString(list):
     '''  
     Creates a list containing the individual bits from a byte-string, integer, 
     or list of values (where each list is interpreted as 1 or 0 based on its 
     truth balue ). 
      
     Bits are ordered from most-significant to least-significant. 
      
     >>> import pcs  
     >>> x = pcs.BitString(0xDEADBEEF)   
     >>> x 
     [1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1] 
     >>> x[4:12] 
     [1, 1, 1, 0, 1, 0, 1, 0] 
     >>> int (x) 
     3735928559L 
     >>> hex(x) 
     '0xdeadbeefL' 
     >>> hex(x[16:32]) 
     '0xbeef' 
     >>> bin(int (x)) 
     '0b11011110101011011011111011101111' 
     >>> str (x) 
     '\xde \xad \xbe \xef ' 
     >>> x[0:8] = 0xAA 
     >>> x[8:16] = [1,1,1,1,0,0,0,0] 
     >>> hex(x) 
     '0xaaf0beefL' 
     >>> int (x) == 0xAAF0BEEF 
     True 
     ''' 
  
     def  __init__(self,bytes): 
          # Convert integer values to their byte representation 
         if  isinstance(bytes,int) or  isinstance(bytes,long): 
             packString = ''  
             if  abs(bytes) <= 0xff : 
                 packString = '!B'  
             elif  abs(bytes) <= 0xffff : 
                 packString = '!H'  
             elif  abs(bytes) <= 0xffffffff : 
                 packString = '!I'  
             bytes = struct.pack(packString, bytes) 
          
          # Convert a list into a byte representation by taking each item 
          # and turning it into a bit. 
         if  isinstance(bytes,list) or  isinstance(bytes,tuple): 
             self.length = len(bytes)*8  
             for  index,i in  enumerate(bytes): 
                 self.append(1  if  i else  0 )   
          
          # For strings or string'ized integers, do bit-shifty stuff 
          # >>> bin(0xFF00) 
          # '0b1111111100000000' 
         if  isinstance(bytes, str): 
             self.length = len(bytes)*8  
              # Do each byte in-order 
             for  i,byte in  enumerate(bytes): 
                  # Most-significant-bit first 
                 for  bit in  range(8 )[::-1 ]: 
                     self.append(1  if  ord(byte) & (1  << bit) else  0 ) 
              
                      
     def  __int__(self): 
         intVal = 0  
         for  index,i in  enumerate(self):       
             intVal += self[-index-1 ] << index 
         return  intVal 
      
     def  __str__(self): 
         value = int(self) 
         formatString = "%0"  + ("%s"  % (len(self)/8 )) + "x"  
         hexstring = formatString % value 
         while  len(hexstring) % 2  != 0 : 
             hexstring = '0'  + hexstring   
         byteString = binascii.unhexlify(hexstring) 
          
         while  len(byteString) < len(self)/8 : 
             byteString = "\x00"  + byteString 
         return  byteString        
      
     def  __hex__(self): 
         return  hex(int(self)) 
      
     def  __getitem__(self, x): 
         return  list.__getitem__(self,x) 
      
     def  __getslice__(self,i,j): 
         return  BitString(list.__getslice__(self,i,j)) 
      
     def  __setitem__(self, i, x): 
         x = 1  if  x else  0  
         list.__setitem__(self,i,x) 
          
     def  __setslice__(self,i,j,x): 
         bs = BitString(x) 
         list.__setslice__(self,i,j,bs) 
      
     def  __invert__(self): 
         return  BitString([0  if  x else  1  for  x in  self]) 
          
     def  __add__(self,x): 
         return  BitString(list.__add__(self,x))

Untitled

2009-07-23T22:52:00.001-04:00

asdfasdf

class  BitString(object): 
     '''  
     Allows bit-level manipulation of a string as if it were an array. 
      
     Example: 
     >>> x = pcs.BitString('\xff \xff ') 
     >>> print x 
     0b1111111111111111 
     >>> print repr (x.bytes) 
     '\xff \xff ' 
     >>> x[4:12] = 0 
     >>> print x 
     0b1111000000001111 
     >>> print bin(x[:8]) 
     0b11110000 
     >>> print bin(x[8:12]) 
     0b0 
     >>> print bin(x[12:]) 
     0b1111 
     >>> print repr (x.bytes) 
     '\xf0\x0f' 
     '''       
     def  __init__(self, bytes): 
         self.value = 0  
         self.length = len(bytes)*8  
         for  i,byte in  enumerate(bytes[::-1 ]):              
             self.value += ord(byte) << (i*8 ) 
     def  __getitem__(self, n):          
         if  n > len(self): 
             raise  IndexError 
         return  1  if  self.value & (1  << (len(self)-n-1 )) else  0  
     def  __getslice__(self, i, j): 
         retVal= 0  
         i = max(i,0 ) 
         j = min(j,len(self)) 
         for  n,bit in  enumerate(range(i,j)): 
             retVal <<= 1  
             retVal += self[bit] 
         return  retVal 
      @property 
     def  bytes(self): return  self.getBytes() 
     def  getBytes(self): 
         formatString = "%0"  + ("%s"  % (len(self)/8 )) + "x"  
         hexstring = formatString % self.value 
         byteString = binascii.unhexlify(hexstring) 
          
         while  len(byteString) < len(self)/8 : 
             byteString = "\x00"  + byteString 
         return  byteString          
     def  __len__(self): 
         return  self.length 
     def  __str__(self): 
         return  self.bin() 
     def  __repr__(self): 
         return  self.__class__.__name__ + "(%s)"  % repr(self.getBytes()) 
     def  __setitem__(self, n, x): 
         x = 1  if  x else  0  
         if  x: 
             self.value |= (x << (len(self)-n-1 )) 
         else : 
             self.value = self.value & self._makeMask(n) 
     def  __setslice__(self, i, j, x): 
         i = max(i,0 ) 
         j = min(j,len(self)) 
          
         isAList = True  
         if  isinstance(x,int) or  isinstance(x,long) or  isinstance(x,bool): 
             isAList = False  
          
         for  index,n in  enumerate(range(i,j)): 
             if  isAList: 
                 self[n] = x[index] 
             else : 
                 self[n] = x 
     def  _makeMask(self, bit): 
         mask = 0  
         bitSkip = (len(self)-bit-1 ) 
         for  i in  range(len(self)): 
             if  i == bitSkip: 
                 continue  
             mask += 1  << i 
         return  mask 
     def  hex(self): 
         return  hex(self.value) 
     def  bin(self): 
         return  bin(self.value)

When used in conjunction with pcs.Field, I've set it up to calculate the value with the new method, but still *use* the value from the old method, to show the difference between the new vs. old method. Pay particular attention to syn, ack, fin, reset, and push.

>>> synAckPkt = tcp('09\xd4191\xf6\xae\x00\x00\x00\x01`\x12\xff\xff\xfe"\x00\x00')
Decoding sport --> 12345
Decoding dport --> 54321
Decoding sequence --> 959575726
Decoding ack_number --> 1
Decoding offset --> 6
Decoding reserved --> 0
Decoding urgent --> 0
Decoding ack --> 1
Decoding psh --> 0
Decoding reset --> 0
Decoding syn --> 1
Decoding fin --> 0
Decoding window --> 65535
Decoding checksum --> 65058
Decoding urg_pointer --> 0
>>> print repr(synAckPkt)
<class tcp {reset: 4, psh: 2, reserved: 0, sequence: 959575726, ack: 1, checksum: 65058, offset: 6, syn: 9, urgent: 0, window: 65535, ack_number: 1, dport: 54321, sport: 12345, fin: 0, urg_pointer: 0}>
>>> print repr(synAckPkt.data)
<class payload {payload: ''}>

Just Kooky

2009-07-22T16:37:00.003-04:00

So PCS can pick up the fields just fine, as long as they're zeros. Make SYN *and* ACK set to '1' and everything is FUBAR.

Packet data taken from:
http://support.microsoft.com/kb/172983

Syn Packet Data:
TCP: ....S., len: 4, seq: 8221822-8221825, ack: 0, win: 8192, src: 1037
dst: 139 (NBT Session)

Syn-Ack Packet Data:
TCP: .A..S., len: 4, seq: 1109645-1109648, ack: 8221823, win: 8760,
src: 139 (NBT Session) dst: 1037

>>> from pcs.packets.tcp import tcp
>>> import binascii
>>> synAck = "008B040D0010EE8D007D747F60122238012D0000020405B42020"
>>> syn = "040D008B007D747E0000000060022000F2130000020405B42020"
>>>
>>> synPkt = tcp(binascii.a2b_hex(syn))
>>> synAckPkt = tcp(binascii.a2b_hex(synAck))
>>>
>>> print repr(synPkt)
[TCP: reset: 0, reserved: 0, sequence: 8221822, ack: 0, checksum: 61971, offset: 6, syn: 1, urgent: 0, window: 8192, push: 0, ack_number: 0, dport: 139, sport: 1037, fin: 0, urg_pointer: 0]
>>>
>>> print repr(synAckPkt)
[TCP: reset: 4, reserved: 0, sequence: 1109645, ack: 1, checksum: 301, offset: 6, syn: 9, urgent: 0, window: 8760, push: 2, ack_number: 8221823, dport: 1037, sport: 139, fin: 0, urg_pointer: 0]

PCS Bit Encoding

2009-07-21T15:05:00.002-04:00

Just some interesting code that I found in PCS. As far as I can tell, the bits of code are the exact same, except for the "shift" field, which is not even used in the scope of the conditional.


            # These two bits of code are the same.
            elif fieldBR > byteBR:
                print "Cannot fit field into current byte"
                shift = fieldBR - byteBR
                mask = 2 ** byteBR - 1
                value = (ord(bytes[curr]) & mask)
                fieldBR -= byteBR
                byteBR = 8
                curr += 1 # next byte
            elif fieldBR == byteBR:
                mask = 2 ** byteBR - 1
                value = ord(bytes[curr]) & mask
                fieldBR -= byteBR
                byteBR = 8
                curr += 1 # next byte
            real_value += value << fieldBR

Yeah

2009-07-21T14:58:00.002-04:00

Working on the aforementioned PCS bug brought to my attention that I've fixed a few bugs in PCS that need to be fixed in the general distribution before running the below script will work.

Rather, try this (which should work in PCS-0.5 without modification):

from pcs.packets.tcp import tcp

t = tcp('09\xd4191\xf6\xae\x00\x00\x00\x01`\x12\xff\xff\xfe"\x00\x00')

print repr(t)

PyDev

2009-07-21T14:41:00.000-04:00

Just found out that PyDev has a built-in Python shell that does completion.

FML.

Full-Stop

2009-07-21T09:41:00.006-04:00

Ran into a BIG issue with PCS last night. Evidently it's mis-identifying packet fields for TCP.

Demo included below:

from pcs.packets.localhost import *
import binascii
x = '020000004500002cca844000400600007f0000017f0000033039d4313931f6ae000000016012fffffe220000'
bytes = binascii.a2b_hex(x)
l = localhost(bytes)
ip = l.data
tcp = ip.data
print repr(tcp)
print tcp

Prints out the following. Notice that instead of SYN/ACK, it is identified as SYN/ACK/RST/PSH.

[class tcp {reset: 4, psh: 2, reserved: 0, sequence: 959575726, ack: 1, checksum: 65058, offset: 6, syn: 9, urgent: 0, window: 65535, ack_number: 1, dport: 54321, sport: 12345, fin: 0, urg_pointer: 0},

class payload {payload: ''}]

That packet is a loopback-ipv4-tcp packet, should look like the following:

Transmission Control Protocol, Src Port: italk (12345), Dst Port: 54321 (54321), Seq: 959575726, Ack: 1, Len: 0

Here's the bytes printed in an easier-to-read format. Emphasis on the flags mine.

02 00 00 00 45 00 00 2c ca 84 40 00 40 06 00 00

7f 00 00 01 7f 00 00 03 30 39 d4 31 39 31 f6 ae

00 00 00 01 60 12 ff ff fe 22 00 00

Interesting Observation

2009-07-17T21:13:00.000-04:00

In Python, None is so truly equivalent to *nothing* that help(None) spawns the interactive help.

I thought that was interesting.

Got Money, Got Paid

2009-07-17T20:43:00.001-04:00

Finally!

Anyway, I haven't mad a blog post in waaaaaaaaaaaaaay too long. Anyway, the bit that I'm working on now is to resolve the issue of sequence number wrap-around. The way that I'm implementing it will (hopefully) have the benefit of automating the calculation of SND.NXT and a buffer of octets that need to be retransmitted based off of the send/sent segments, and SND.UNA.

Essentially, it's a continuous list with an offset. Less simple than it would originally appear due to the way slices work. Here's a quick example:

Example:
>>> from segmentBuffer import *
>>> sb = segmentBuffer(base=10, max=15)
>>> sb += range(8)
>>> sb[10] # The first item is at [10]
0
>>> sb[11] # The second item is at [11]
1
>>> sb # However, the items are in the proper order
[0, 1, 2, 3, 4, 5, 6, 7]
>>> sb[10:] # And are properly sliced
[0, 1, 2, 3, 4, 5, 6, 7]
>>> sb[10:2] # And the slices wrap-around at max
[0, 1, 2, 3, 4, 5, 6]
>>> sb[2:10] # And slices stop when it hits a gap.
[7]
>>> sb == sb[10:] # Truth works just fine.
True
>>> sb += 2 # The offset is incremented easily
>>> sb # Notice that the first two items are deleted
[2, 3, 4, 5, 6, 7]
>>> len(sb) # The length indeed goes down 8->2
6
>>> sb += range(100,200)
>>> len(sb) # Adding 100 items keeps the correct length
15
>>> sb # Only items up to the limit are added
[2, 3, 4, 5, 6, 7, 100, 101, 102, 103, 104, 105, 106, 107, 108]
>>> sb2 = sb + range(300,400)
>>> sb is sb2 # Addition operator returns a new object.
False
>>> sb2 += 3 # The other object can be incremented
>>> [i for i in sb if i not in sb2]
[2, 3, 4] # Just an example to show the difference
>>> sb[12:15] # This shows that the items were taken off the front
[2, 3, 4]
>>> sb[12:2]
[2, 3, 4, 5, 6]
>>> sb2[12:2] # The indexes for the remaining items do not change
[5, 6] # Although the deleted items are gone.
>>> sb
[2, 3, 4, 5, 6, 7, 100, 101, 102, 103, 104, 105, 106, 107, 108]
>>> sb2
[5, 6, 7, 100, 101, 102, 103, 104, 105, 106, 107, 108]

Birthday and Eclipse Galileo

2009-07-09T20:15:00.001-04:00

The birthday went pretty well (I'm still alive, after all) and I just grabbed Galileo and installed all of my tools (PyDev, CDT, Perforce) that I need on a regular basis. Excited to see that there's a Cocoa implementation of Eclipse now, and Galileo seems to start up a lot faster than Ganymeade (don't know if that's an effect of Cocoa-Carbon or general bloat reduction).

WTF

2009-07-06T22:01:00.001-04:00

Evidently Ecto ate my last SEVERAL blog posts, by not actually posting anything.

Hmm

2009-07-06T01:43:00.000-04:00

No, no, no

2009-07-06T01:07:00.000-04:00

Okay, okay, okay

2009-07-05T23:33:00.000-04:00

I Think I've Got It!

2009-07-05T22:26:00.003-04:00

I might have finally figured out the packet flow for the most flexibility and utility.

Packet arrives on interface
Is queued by libpcap
Thread grabs packet off of pcap queue
IF the packet is NOT the packet we are expecting next (seg.sequence != rcv.nxt)
1. Throw it into a list used by the thread.
2. Go back up one level
ELSE the packet is one we were expecting
1. Add the packet to the State Machine's recv() queue, IF there is not already a packet with the same sequence number
2. IF processing is enabled
  1. Process the packet (i.e. "Segment Arrives" stuff)
  2. Advance the 'processed packet' index, which points to the last packet in the recv() queue that has been processed.
3. Iterate over all packets in the thread's personal queue. This is done because we might have already received the 'next' packet, but it was received out-of-order.

Okay, so now the packets are being added to a queue, and they are in the correct order. They are also conditionally processed/handled/whatever you want to call it.

Now, we are writing a test. We want to stop when [1] A certain packet is received or [2] A specific state change occurs. This means that two methods need to be hooked into: the state-change method, and the packet-handling method. With the state change, we will want to pass control to the user (i.e. disable auto-processing) AFTER the state transition. With the packet-recv stuff, we want to do it BEFORE the packet is processed. This means that there needs to be a step between '5.1' and '5.2' that checks to see if the packet is flagged, and appropriately disables processing when the packet is encountered.

The test-writer then can call the 'recv()' method and be guaranteed to get the next packet, in the proper order. Additionally, the test-writer can set "break points" that disable auto-processing. This is important, because auto-processing can lead to sending out additional packets (i.e. response to a SYN-ACK, response to a FIN packet, etc.) that may need to be omitted for a test to be successful.

Whew.

Multiprocessing

2009-07-04T02:51:00.001-04:00

Hmm. Multiprocessing may not be the way to go, as it runs into issues.

Might have to think about this some more.

Implementing TCP Stack Stuff

2009-07-04T00:43:00.001-04:00

So I'm implementing more and more of the TCP stack. The idea is that if I actually implement a TCP stack (which was expected, in small chunks) then I [1] understand how TCP should operate much better which allows me to [2] write tests much better and [3] identify corner-cases much better.

One of the issues that I ran into tonight reaches back to my idea to run a separate process that's always "listening" for new packets, and does all of the "Segment Arrives" processing (see pp65 in the RFC).

In order to manipulate the TCP stack of some remote host, it is expected that it will be necessary to get one side of the connection into a specific state (e.g., get the remote state to be SYN-SENT, or the local state to be SYN-RECEIVED) and then perform specific operations that test very specific points of the specification.

In order to do this efficiently, it becomes of the utmost importance to be able to get into these states very easily. Connection buildup and tear-down must be simple, as they are not important except for the individual tests that test these portions (and even then, we might want to skip right to receiving the SYN-ACK packet, before sending the ACK packet to enter the Established state).

In order to perform these state transitions gracefully and properly, it becomes necessary to implement the entire TCP stack, and build in hooks that halt at certain points (on a state transition, specific packet sent or received, malformed packet, etc.).

Now, it turns out that I still hadn't done enough planning to foresee the issue with packet ordering. The model that I thought would work would be to call a "recv()" method that would receive the next packet, and then the test would do something interesting with it. That becomes a problem when the test is expecting a specific field -- say, with a specific ACK number -- but receives that packet after receiving some other packet. The test might not properly realize that the first packet (which didn't match the expected pattern) was pertinent, and would be needed later in the test.

This gives two options:

The recv() method should only return the next packet (i.e. the segment's sequence number == rcv.nxt)
The recv() method simply fetches the next packet off of a queue, to which packets are added in-order by a separate thread/process that manages the state.

Both of these approaches have their limitations. Consider a test in which we want to send a malformed ACK packet in response to a SYN-ACK packet. The code for that might look something like this:

tsm = TcpStateMachine()
i = tsm.packetsRecvd.size()
syn = tsm.newPacket({'syn':1})
tsm.sendRawTcp(syn)
startTime = time.time()
# Keep looping. Note that this loop would be rolled into a seperate method.
while True:
  # 30 second timeout. Test fails.
  if time.time() < startTime + 30:
    return False
  # If no new packets we received, sleep
  if tsm.packetsRecvd.size() <= i:
    time.sleep(0.1)

  # Loop through available packets.
  for i in range(i, tsm.packetsRecvd.size()):
    p = tsm.packetsRecvd[i]
    # See if it's a SYN-ACK and the ack number matches.
    if p.ack and p.syn and p.ack_number == syn.sequence + 1:
      break
  # Do stuff with packet at offset 'i'.

The code would send a SYN packet, with the other fields auto-filled, then wait until some packet is received, then do processing based on that.

Actually, after writing that, performing some method of callback on a state transition could lead to un-necessary complications. What might be more useful is a way to turn off auto-processing (not receipt of, just processing of) packets upon some trigger (i.e. after transitioning to state 'FIN-WAIT-1', don't do anything with the packets), and a way to see if that trigger has been hit (although checking to see if the state is FIN-WAIT-1 should be sufficient).

Hmm.

Working all of this out in writing might have cleared things up a bit.

Thanks!

Time Off Did The Trick

2009-07-02T21:51:00.001-04:00

Looks like a little time-off did the trick. Got back into the problem that I was working on last (fighting with the operating system, and the various interfaces). Looks like using 127.0.0.X where X is not '1' works to elicit the correct SYN/ACK response, on the correct interface (lo0), and the host OS doesn't try to fight me about it.

Now all I've gotta do is get the test working :-)

Hopefully, things will be smooth sailing from here on forward!

Soy Libre!

2009-07-02T19:26:00.001-04:00

Just finished up the last of my exams. About friggin' time. I feel pretty confident about all of them -- and I've been doing better this semester than a usually do.

Schedule for the rest of the evening is GSoC, same for tomorrow. Saturday I've got a BBQ for the 4th. GSoC starts accepting status reports on the 6th, they're due on the 13th (which means I have to submit it to Google/GNN, and they/he has to approve it by then).

Should be an interesting weekend.

I turn 21 next Wednesday (the 8th) so count on nothing getting done on the 8th/9th (Maybe on the 9th, I took the day off from work).

Also

2009-06-28T21:30:00.001-04:00

Enabling "stealth mode" (which should kill all responses to an unsolicited [from the OS's point of view] SYN/ACK packets) does not prevent the OS from responding on lo0. Evidently stealth mode is not active on the loopback interface.

Grrrrrrr.

Power is back on, But network problems continue

2009-06-28T21:01:00.001-04:00

Power's back on, woooooooooo.

Okay, so I have two problems:
[1] If I attempt to 'send' packets 'from' an IP address that my computer actually has, the host OS's TCP stack sends a RST packet when it sees a SYN/ACK response to my 'sent' SYN packet.
[2] If I attempt to 'send' packets 'from' a different IP address, the packets get bounced back and forth several times between my computer and the router. For whatever reason, the listening server ('nc -l <port>') doesn't respond at all.

Yay.

2009-06-28T17:12:00.000-04:00

Power is out. Posting this from my iPhone.

Fighting with the host OS

2009-06-28T14:24:00.001-04:00

After finding out that Wireshark is stupid, and considers 0x2000 to be the same as 0x0002 for Loopback-layer family fields, I am now battling with the host OS. Sending a connection request properly elicits a SYN/ACK response, but the OS is quickly sending out a RST packet to kill the connection.

Changing the source IP to be something other than an IP the operating system should care about results in no SYN/ACK packet. I tried moving back over to a non-loopback interface, with a different source IP address (using the correct destination IP address). It gets sent back from the router (the packet is duplicated with the source MAC being the router MAC), but it does not elicit a SYN/ACK response.

Wierd. And frustrating.

Ah.

2009-06-28T00:32:00.001-04:00

So, evidently even if you have a host that is listening on 0.0.0.0:12345 (e.g. "nc -l 12345") and inject packets onto an active interface, pointed at that interface's IP address (e.g. "en1" has IP 172.16.0.10) -- any normally-generated packets to/from a local IP address go over "lo0".

That would have been good to know, what, a few hours ago?

Troubleshooting

2009-06-27T22:23:00.001-04:00

Lots of troubleshooting the code right now, trying to get the 3-way connect to work. Lots of issues with typos, unfortunately. A lot of it had to do with using (for example) "off" in place of "offset" to access the offset of a TCP packet, because of the way it's declared:

off = pcs.Field( "offset", 4 )

My eyes just saw the first part, and ignored the second half when I started writing code. Thought I went back and fixed all of the fields, evidently not!

P4 Is Getting Kind of Annoying

2009-06-26T00:04:00.001-04:00

I don't know if it's P4 itself, or P4 integration with Eclipse, but a versioning system should never prevent editing local files because a connection attempt failed, or because the host-name of the machine changed.

That is all.

Checksumming No Se Funciona

2009-06-20T21:12:00.001-04:00

Alright, so the last part of the puzzle that needs to fall in place is the checksum. Conveniently, the algorithm for checksumming IP and TCP packets is the same (albeit you feed them different sets of information).

The algorithm works somewhat like so (source IBM):

ushort checksum16(uchar* data, int len)
{
    uint sum = 0;
    if ((len & 1) == 0)
        len = len >> 1;
    else
        len = (len >> 1) + 1;
    while (len > 0) {
        sum += *((ushort*)data);
        data += sizeof(ushort);
        len--;
    }
    sum = (sum >> 16) + (sum & 0xffff);
    sum += (sum >> 16);
    return htons(~sum);
}

That is, it sums all of the 16-bit words, does some 1's compliment fanciness, and then returns the 16 least significant bits (inverted). Of course, the operation can be done with any arbitrary multiple of 16 bits. Since most machines nowadays use 32- or 64-bit arithmetic, there's no reason not to do it with 32 bits (or 64). However, the final result is always 16 bits wide.

In Python, the code looks like (or so I think) the following:

def checksum( bytes ):
    tmpbytes = bytes
    total = 0
    # Must be a number of bytes equal to a multiple of two.
    if len( tmpbytes ) % 2 == 1:
        tmpbytes += "\0"
    # For each pair, add the value to the total.
    for i in range( len( tmpbytes ) / 2 ):
        total += ( unpack( "!H", tmpbytes[( 2 * i ):( 2 * i ) + 2] )[0] )
    # Keep shifting
    while total >> 16:
        total = ( total >> 16 ) + ( total & 0xffff )
    # One's complement
    total = ~total
    # Make into 2 bytes
    total = total & 0xffff
    # Return value
    return total

However, given an IP layer: 0x4500004096da40004006eca1c0a80168d04524e6, the checksum is 0xECA1. When calculating the checksum, the checksum field is set to zero (0x0000). So, the checksum is calculated like so:

>>> from binascii import *
>>> from struct import *
>>> data = unhexlify('4500004096da40004006<b>0000</b>c0a80168d04524e6')
>>> data
'E\x00\x00@\x96\xda@\x00@\x06\x00\x00\xc0\xa8\x01h\xd0E$\xe6'
>>> cksum = checksum(data)
>>> cksum
60577
>>> hex(cksum)
'0xeca1'

Obviously, the end result is incorrect. Turns out that the values that I was using for testing were incorrect. I've since fixed them in the blog post and the offending code.

Glad I got up early

2009-06-19T08:31:00.001-04:00

So I got up early before class today to work on the problem. I don't know what the issue was, I think it was a matter of incorrectly setting the TCP offset that made Wireshark freak out.

Another PCS Architecture Rant
This time, I'm a bit curious about the design of the pcs.Chain class. It inherits from the 'list' base type, but instead of using itself to store the data, it actually *contains* a seperate list. Example:

>>> isinstance(myChain,list)
True
>>> myChain
[]
>>> myChain.packets
[<ethernet: src:="" \x00\x1bc\x06\x82\xb2',="" dst:="" \x00!)\xa5\xa9?',="" type:="" 2048="">, <ipv4: hlen:="" 5,="" protocol:="" 6,="" src:="" 2886729738l,="" tos:="" 0,="" dst:="" 3494107623l,="" ttl:="" 64,="" length:="" 44,="" version:="" 4,="" flags:="" offset:="" checksum:="" id:="" 56649="">, <tcp: reset:="" 0,="" reserved:="" sequence:="" 3915041816l,="" ack:="" checksum:="" 60177,="" offset:="" 5,="" syn:="" 1,="" urgent:="" window:="" 65535,="" push:="" ack_number:="" 0l,="" dport:="" 80,="" sport:="" 52722,="" fin:="" urg_pointer:="" 0="">, <payload:>]
</payload:>

Not sure what's going on with that. Its constructor only takes a list object:

class  Chain(list):
     def  __init__(self, packets = None ):
         list.__init__(self) 
         self.packets = packets 
         self.encode() 
# Versus
class  Chain(list):
     def  __init__(self, packets = None ):
         list.__init__(self, packets)

This would end up saving a lot of time and make the resulting code look cleaner:

That would make it easier to perform nifty operations. pcs.Chain offers a "checksum" operation, that operates on the whole chain. However, consider the utility of "myChain[-2:].calc_checksum()". Now we can checksum an arbitrary range of packets! Not immediately useful, unless you realize that this is a nifty way to calculate the IP checksum ;-).

Also, I have to wonder about the duplication of functionality offered by the pcs.Chain class, and the pcs.Packet.data member. It seems that, for the large part, they perform the exact same function. Consider:

>>> p = payload()
>>> t = tcp()
>>> i = ipv4()
>>> e = ethernet()
>>> e.data = i
>>> i.data = t
>>> t.data = p
# Finally, these two should have the same ultimate result
>>> e.chain()
>>> chain([e,i,t,p])

I can see the reasoning behind the "data" member -- it makes it easy to set-and-forget what the next level is, and the "chain()" method is a nice helper that builds a chain. Consider the earlier example of myChain[-2:].calc_checksum() currently looks like: myChain.packets[-2].chain().calc_checksum().

Malformed Packet Bull@$!#

2009-06-18T08:01:00.001-04:00

Getting pretty sick and tired of Wireshark being useless in telling me exactly why my TCP packets are malformed.
Just sayin'.

Unrelated Note

2009-06-13T22:48:00.001-04:00

On an unrelated note, FreeBSD handles pthread_mutex's shared status properly. Mac OS X does not.

Debugging code that relies on this on Mac OSX may cause hair loss.

(This was from my first CSE410 project, thought I'd post it here while I was posting things).

Bailed on previous idea

2009-06-13T22:09:00.001-04:00

Also of note, I bailed on the idea from last night (and before) of using raw IP sockets to do the sending. The idea was to avoid needing to ever touch the physical or IP layer, thus removing some amount of work (e.g., no setting of the IP length fields, IP checksum calculation).

However, using raw sockets has the caveat of not providing a receive buffer. If using raw IP sockets, one cannot override the 'protocol' field to be IPPROTO_TCP. Haven't put much more thought into using raw TCP sockets.

Short Rant on pcs.Field
The class pcs.Field is used as a base class to represent any field inside of a packet. It provides bit-width flexibility, and the ability to encode/decode values of arbitrary bit-width. This is very useful when defining the various fields of a packet, as you can specify their exact size, and then with the pcs.Layout class, specify their exact order.

However, in order to string these two things together, you get somewhat of a kluge. Consider the following code in tcp.py:

     def  __init__(self, bytes = None ):
          """initialize a TCP packet""" 
         sport = pcs.Field("sport" , 16 ) 
         dport = pcs.Field("dport" , 16 )

And then the corresponding code to manipulate those fields...

>>> from pcs.packets.tcp import tcp
>>> t = tcp()
>>> type(t.offset)
<type 'int'>
>>> type(t.sport)
<type 'int'>

Okay, great. In the __init__ method, they were working on local variables with the same name as the variables to be used. The pcs.Field objects are supplied with a name to correspond to the object property (e.g. the pcs.Field object with (... name="sport",...) corresponds to the actual field "pcs.Field.sport". Great.

However, the approach falls short here. pcs.Field provides no way of setting the bit-order of whatever field it is manipulating, so users are stuck doing this:

t.sport = socket.htons(80)
# Instead of
t.sport = 80

And having the "magic" just happen. This prevents simple comparisons, as well. Which of the following is easier to read?

if ntohs(t.sport) == 80: ...
if t.sport == htons(80): ...
if t.sport == 20480: ...
if t.sport == 80: ...

This is less trivially simple for fields that may have more than one representation. This was discussed in a previous post (IIRC) with relation to IP and Hardware addresses. Consider the following. The IP address "127.0.0.1" can be represented as:

16777343 (host-byte-order)
2130706433 (network-byte-order)
'\x01\x00\x00\x7f' (host-byte-order byte-string)
'\x7f\x00\x00\x01' (network-byte-order byte-string)
And of course, "127.0.0.1"

Obviously, this makes things a bit difficult to work with. However, the current method does not just use network-byte order. An IP address is a NBO long. A port is a NBO short. An ethernet address is a NBO byte-string. Figuring out what the proper internal representation for everything is a pain, and translating between HBO and NBO makes it worse.

The simple solution is to allow multiple ways of getting and setting a field. Example using ports (also works with IP addresses, ethernet addresses):

>>> t.sport.setAscii('80')
>>> t.sport.setInteger(80)
>>> t.sport.setNetworkInteger(20480)
>>> t.sport.setBytes('P\x00\x00\x00')
>>> t.sport.setNetworkBytes('\x00\x00\x00P')
>>> t.sport == 80
True
>>> t.sport.getInteger() == 80
True
>>> t.sport == '80'
True
>>> t.sport == 'P\x00\x00\x00'
True
>>> t.sport == '\x00\x00\x00P'
False
>>> t.sport.getNetworkBytes() == '\x00\x00\x00P'
True

This allows us all the flexibility in the world that is needed. Assuming that all of the different types support the base functions (get/set for ASCII, Integer, and Bytes in both host- and network-byte order), everything is quickly extendible.

Visiting the original code, things have improved slightly. No longer do we need to worry about bit-ordering.

# Old
t.sport = htons(80)
t.sport = htons(str('80'))
t.sport = struct.unpack('!L','\x00\x00\x00P')
# New
t.sport.setInteger(80)
t.sport.setAscii('80')
t.sport.setNetworkBytes('\x00\x00\x00P')

Alright, now we face the problem of making the encoding 'cleaner'. Encoding things is extremely simple if everything is aligned on an 8-bit byte boundary. One could simply use "t.sport.getNetworkBytes() + t.dport.getNetworkBytes()" ad infinitum. However, that approach is ugly. Overriding __add__() with some gentle application of isinstance() and you can get the bytes pretty easily (even without 8-bit byte boundaries).

Will do some work tonight on this front. Will make for a very worthwhile patch to PCS.

[/rant]

If Red Bull would just...

2009-06-13T17:46:00.001-04:00

...sponsor Google Summer of Code, it would make my life much easier (and less expensive).

Works pretty well

2009-06-13T04:17:00.001-04:00

After moving everything to Python 2.6 (read: re-installing Pyrex, PyLint, PyLibPcap, and PCS), the packet-queueing process was extremely easy to do. Much easier than I thought it would be. Here's an example of it in action (note that the TCP data that is interpreted by PCS is whatever the equivalent of the ASCII data is).

Note that the output gets a little bit wacky as both processes fight for StdOut.

Python 2.6.2 (r262:71600, May  2 2009, 17:25:25)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from packetQueue import *
>>> ipl = IpListener("127.0.0.1")
>>> ipl.process.start()
>>> 
>>> from socket import *
>>> host = gethostbyname(gethostname())
>>> port = 0
>>> text = "Hello"
>>> 
>>> def send(data=text):
...   sender = socket(AF_INET,SOCK_RAW,IPPROTO_IP)
...   sender.sendto(data, (host,port))
... 
>>> send()
>>> send(Hello
'This is a test')
>>> 
>>> ipl.recvQueue.get()
This is a test
<TCP: reset: 0, reserved: 0, dport: 27756, sequence: 1862270976, ack: 0, checksum: 0, ack_number: 0, syn: 0, urgent: 0, window: 0, offset: 0, push: 0, sport: 18533, fin: 0, urg_pointer: 0>
>>> ipl.recvQueue.get()
<TCP: reset: 29, reserved: 13, dport: 26995, sequence: 543781664, ack: 7, checksum: 0, ack_number: 1629516901, syn: 58, urgent: 3, window: 0, offset: 7, push: 14, sport: 21608, fin: 0, urg_pointer: 0>
>>> 
>>> ipl.process.terminate()

First Round Done

2009-06-13T01:21:00.001-04:00

Not really a project update, but a worthwhile note: the first round of exams are done at MSU, and I have already completed my second-semester project for my CSE course. There's a paper due on the 25th, and the homework for all of the next week is already done. Should be able to put in some extra time this weekend and the upcoming week. Right now is about 3 weeks from the end of the semester, then my it's hyperspeed.

Although...

2009-06-11T02:20:00.001-04:00

Although there IS the catch that if we do not call recv() before the response TCP packet arrives, it will be lost forever (since we are using raw sockets). The easy solution to this is to have a thread that does all of the buffering, as I imagine happens in a real TCP stack. Might have to look into the 'multiprocessing' module. It seems like passing pcs.packets.tcp.tcp objects via a multiprocessing.queue would be the way to go about doing things.

Actually, after reading a bit about the multiprocessing module, this may end up being a non-issue. Just a have a reader thread that reads all of the raw IP data, checks to see if it is the right stuff (i.e. the IP addresses are what were expected, the TCP ports are the expected ports) and then passes it along to the queue. Behind-the-scenes, a call to recv() will just fetch the next item off the queue.

Working with Raw IP

2009-06-11T02:13:00.001-04:00

I figure that messing with IP stuff at any point in time is a Bad Idea (TM), so doing anything at all with the pcs IP or Ethernet layer shouldn't be done. Ideally, I should be able to point an IP packet (filled with TCP data and payload) at some IP address, and everything will get sorted out.

I figure it'll work something like the code below (need root, open two Terminals, call recv() before send()). Not sure that much more robustness is needed...

from socket import *
host = gethostbyname(gethostname())
port = 0
text = "Hello"
def send():
sender = socket(AF_INET,SOCK_RAW,IPPROTO_IP)
sender.sendto(text, (host,port))
def recv():
recver = socket(AF_INET,SOCK_RAW,IPPROTO_IP)
data = recver.recv(1024)
print data[20:]

Sexy

2009-06-07T17:06:00.001-04:00

Is what pylint is.
Integration with PyDev (Eclipse) makes it even better.

Hi, George

2009-06-04T21:44:00.001-04:00

George

Looks like there's been an issue again...

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

YOU@neville-neil.com

Message will be retried for 2 more day(s)
...
For help, please quote incident ID 13500304. (state 18).

Logic Validation

2009-05-31T05:50:00.003-04:00

Looking for some input on folks familiar with low-level TCP. I've got some general questions about proper values of Sequence # and Ack # over the course of a TCP session.

ACKing Packets

A segment on the retransmission queue is fully acknowledged if the sum of its sequence number and length is less or equal than the acknowledgment value in the incoming segment.

A packet is received with sequence number "seqn" and TCP payload with "len" octets of data. The receiving party should compare "seqn" to the variable outlined in the RFC, "rcv.nxt". If they match, "rcv.next" is incremented by "len".

The ACK number of outgoing packets is always sent to 'rcv.next'-1.

What is done with packets received out-of-order, but that are entirely valid? Buffer them, then check each one's SEQ number when rcv.nxt is incremented?

Delayed ACK response
This also raises the question of how to properly ACK a received message, when packets aren't already being set. In some instances, it may be necessary to send an empty packet with the ACK flag set. How to determine when this needs to be done?

IP and Port Representation

2009-05-31T05:50:00.001-04:00

Small hurdle that I tried to tackle tonight is the representation of IP addresses and ports. There are at least 4 representations of each that will need to be dealt with regularly:

- ASCII (e.g. "127.0.0.1", "80")
- Host-byte-order integer, useful for ports (16777343, 80)
- Network-byte-order integer, used in raw packets (2130706433, 20480)
- Network-byte-order byte-string, used by "inet_aton" and the like. ('\x7f\x00\x00\x01', '\x00\x00\x00P')

So, the IP address and Port were encapsulated in classes. This should make dealing with them slightly easier.

Oh.

2009-05-31T02:41:00.001-04:00

I was having some issues earlier wrapping my mind around why, exactly, packing strings didn't give me a string with the "\x##\x##\x##" format. Evidently, that is only necessary for characters that do not have a normal, printable equivalent.

Example:

>>> "\x7D" + "\x7B"
'}{'

Anyway, the following output is what had me confused, and why I spent about an hour or so now looking into the 'struct' package and why it wasn't giving me what I thought I wanted. Notice the plaintext "test" at the end. It seemed out of place, so I was assuming that I was doing something wrong.

>>> from pcsextension.payload import payload
>>> from pcs.packets.tcp import tcp
>>> p = payload('test')
>>> t = tcp()
>>> t.data = p
>>> t.chain().bytes
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00test'

Rejected Email

2009-05-31T01:25:00.001-04:00

George

If you're reading this, I'm having some issues contacting you via email:

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

gnn@neville-neil.com

Message will be retried for 2 more day(s)

Technical details of temporary failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 451 451 4.3.0 Message held for human verification before permitting delivery. For help, please quote incident ID 13393043. (state 18).

Memorial Day

2009-05-25T14:51:00.001-04:00

Happy Memorial Day, All!

Logging Framework Change

2009-05-24T19:50:00.000-04:00

Decided I was going about it all wrong (it was getting late). Changed the implementation to simply provide convenience methods on top of the Python 'logging' package's framework -- with the added benefit of not having to worry about the nitty-gritty of the 'logging' package. Just simple, direct access to the log methods.


     >>> from loggable  import Loggable
     >>> class A(object): 
     ...    def  __init__(self): 
     ...      self.log = Loggable (self) 
     ...   
     >>> a = A() 
     >>> a.log.info('test') 
     2009-05-24 12:48:08,654 - __main__.A - INFO - test 
     >>> a.log.critical('test') 
     2009-05-24 12:48:19,475 - __main__.A - CRITICAL - test 
     >>> a.log.state('test') 
     2009-05-24 12:48:22,978 - __main__.A - STATE_CHANGE - test 
     >>> a.log.pktsent ('test') 
     2009-05-24 12:48:26,402 - __main__.A - PACKET_SENT - test 
     >>> a.log.generated('test') 
     2009-05-24 12:48:49,516 - __main__.A - RESPONSE_GEN - test 
     >>> a.log.field('test') 
     2009-05-24 12:48:52,475 - __main__.A - FIELD_CHANGE - test

Also have been working on a simple TCP State Machine today. My theory is that we will need to know [1] what state the local TCP connection is in, and [2] what state the remote TCP connection should be in. A simple state machine should allow us to do this with some flexibility. Here is the base class/interface. Suggestions are highly welcomed:

class  TcpState(object):
    ''' 
    Base TCP State class.   Note that it will, by design, ignore all non-TCP layers.
    The TCP State classes will not perform verification that the packets are all
    to/from the same hosts.   That filtering will be implemented elsewhere in the
    framework.
    '''
    
    def  __init__(self):
        self._nextState = None 
    
    def  packetSent(self, tcpPacket):
        ''' 
        Hand off a packet to the TcpState object, where the object will perform
        any necessary calculation, and determine if a state change should be
        caused by the provided packet.
        
        The packet should be a packet that is considered 'sent' from
        the TCP stack that this state belongs to.
        
        @return True if the packet causes a state change.   False otherwise.
        '''
         return  False
    
    def  packetRecvd(self, tcpPacket):
        ''' 
        Same as packetSent, but in the opposite direction.
        
        The packet should be a packet that is considered 'received' from
        the remote TCP stack.
        
        @see packetSent
        '''
         return  False
    
    def  causedStateChange(self):
        ''' 
        Returns true if a previous call to packetSent/packetRecvd caused a  
        state-change.
        '''
        return  self._nextState != None 
    
    def  getNextState(self):
        ''' 
        Returns the class (subclass of TcpState) that represents the state that
        the TCP stack should be in, as a result of previous packetSent/Recvd  calls.
        @return A class object, or None if no state change should occur.
        '''
        return  self._nextState

Interesting Predicament: Blogo does not like pasting HTML code inside of "pre" tags. It tries to make the HTML code visible, by changing each < and > to "<" or ">", respectively. It seems that I can't find a Blog client that does everything perfectly. So no Python syntax highlighting for now.

Logging Framework

2009-05-24T03:38:00.001-04:00

I think I've got the Logging stuff set up the way I'd like it to work.

Here's an example (the doc for the responsible class)...

    >>> from loggable import Loggable
    >>> Loggable().logfield('test')
    2009-05-24 03:23:40,851 - loggable.Loggable - FIELD_CHANGE - test
    >>> Loggable().loginfo('test')
    2009-05-24 03:24:26,108 - loggable.Loggable - INFO - test
    >>> class B(Loggable):
    ...   pass
    ... 
    >>> B().loginfo('test')
    2009-05-24 03:24:38,306 - __main__.B - INFO - test 
    >>> B().logpacket('packet XYZ sent')
    2009-05-24 03:26:05,299 - __main__.B - PACKET_SENT - packet XYZ sent
    >>> B().logresponse('response ABC generated for packet XYZ')
    2009-05-24 03:26:15,745 - __main__.B - RESPONSE_GEN - response ABC generated for packet XYZ
    >>> B().logfield('field abc123 set to newValue')
    2009-05-24 03:26:42,292 - __main__.B - FIELD_CHANGE - field abc123 set to newValue
    >>> Loggable().logstate('state changed to CONNECTED')
    2009-05-24 03:31:16,848 - loggable.Loggable - STATE_CHANGE - state changed to CONNECTED

Further thoughts on design

2009-05-24T01:08:00.001-04:00

A few more thoughts on the design of the framework:

Logging (textual)
Logging (packets)
Diff'ing (packets)
Live history

Logging (textual)
The way that the textual output of the framework is delivered needs to be well-thought-out. While it's obvious that "test passed" / "test failed" is all that should be necessary for standard runs (INFO level), there are many distinct levels below INFO that will be necessary. A few of these off the top of my head:

Full packet sent'd/recv'd over network IF
This is obvious. Just a Packet sent/recv, with the BinHex print-out
State-change
This should somewhat reflect the diagram provided on Page 23 of RFC 793. This behavior will also make it easier to track issues in the flow.
1. Closed
2. Listen
3. SYN sent
4. SYN recvd
5. Established
6. Fin Wait-1/2
7. Close Wait
8. Closing
9. Time Wait
10. Closed
Response generation
Automated response generation (i.e. Send packet X, recv packet Y. Validate that Packet Y is what we were expecting, and then generate Packet Z that is a valid response/ACK for Packet Y).
Field generation (dynamic)
When sending information, some fields (like IP length, checksums) may be generated dynamically/on-the-fly.
Manual field override
Testing invalid states/data will also be necessary. Manually overriding fields (e.g. bad checksum) should provide extra insight. This might belong on the same logging level as Field Generation, but with a different prefix.

Logging (Packets)
Creating a TCPDUMP-formatted file with the contents of all of the packets from a test session will be invaluable in determining what, specifically, went wrong during the tests. This will make the use of third-party tools possible, and should complement the configurable logging well.

Differential Analysis
Simple packet replay, will only be useful in select circumstances. Ideally, sending packet X will always generate response Y (with some variables, such as timestamp). Playback of a TCPDUMPed session would allow this, but it would need a framework to determine what fields are important for validity, and which aren't. Since it would be difficult to require the specific lack of a particular response, this application may be very limited.

Live History
Having a history (in memory) of all of the packets sent/received during the current test will be a necessity. This is required to make sure that, for example, all of the packets were properly ack'ed as we continue sending packets. Also, it will be necessary to have internal state information about how much data has been ack'ed, and its correlation to the window size.

Not Just OS X

2009-05-23T20:21:00.001-04:00

The reason for my previous post, was to ensure that the issues that I encountered building the Hg-current build of PCS were not a result of running on Darwin/OSX instead of FreeBSD. This worry has been alleviated - the same issues occur on FreeBSD.

I Love X

2009-05-23T20:15:00.001-04:00

I just wanted to note that X11, for its age, is amazing. X11 + SSH with X Forwarding + VMWare Fusion + OS X for the win. Saves me from having to run a full GUI inside the virtual machine.

(Of course, it wouldn't be necessary if FreeBSD shipped with links, lynx, or wget. I understand that there's licensing issues, and that it comes with package management and curl.)

PCS-Current

2009-05-23T16:12:00.001-04:00

Working on getting the latest PCS from Hg working, because it will be necessary to have proper Checksum and Length generation on-the-fly for easy generation of valid packets.

Contacted GNN with (hopefully) the last few issues to getting this to work for me.

Zach

Wireshark Behavior

2009-05-22T23:01:00.001-04:00

Looks like the issue is the adapter.

http://www.mail-archive.com/wireshark-users@wireshark.org/msg03669.html

Wireshark Strangeness

2009-05-22T22:46:00.001-04:00

Playing around with Wireshark, looking at the dissected data, and getting some weird results. Evidently the small echo server I wrote (see below) does not create the correct TCP checksum. I thought that this might be an issue when using lo0 (loopback), but the data simply doesn't show up in Wireshark when going to/from the same IP address (in my case, 172.16.0.10). Shows up in netstat:

tcp4       0      0  172.16.0.10.46002      172.16.0.10.55074      ESTABLISHED
tcp4       0      0  172.16.0.10.55074      172.16.0.10.46002      ESTABLISHED
tcp4       0      0  *.46002                *.*                    LISTEN

Not sure what its issue is.

Loopback pcap file available here.

Structure

2009-05-22T18:01:00.001-04:00

Thinking about the best way to structure the TCP regression framework. A few things immediately jump out at me:

Handle either IPv4 or IPv6 gracefully
Reuse the most code possible
Easy generations of responses (e.g. create new Chain, with corresponding TCP layer with filled-in fields like Sequence number, Ack number, Ack flag, etc.)
Ability to test that the opposite side is actually getting a connection and recv'ing data

Handling IPv4 and IPv6 Gracefully

The simplest way to do this would be to define an interface and two implementations. However, I need to look into the differences between IPv4 and IPv6 a bit more -- static header size may preclude IPv6 from needing some tests, and there may yet be other differences.

Code Reuse

Code reuse will likely fit into the same category as above. Simple interface/implementation class hierarchy may do the trick. However, aside from the direct manipulation of the tests, there may be some tests which are a subset of other tests, or the same test may need to be run under multiple circumstances. Preventing duplicated code is something I really put a high priority on.

Response Generation

Response generation will also be a subset of the two above areas. For example, let's say I send a SYN packet, and I find the corresponding SYN/ACK packet. Having a method that generates the final ACK packet would be beneficial (or that generates an ACK packet to any packet).

Connectivity Testing

Connectivity testing should be fairly trivial. Basically, set up an echo server. If the connection is properly established, the data should be echoed back without problem. This particular solution should also lend itself to testing window sizes.

Side Note:
The blog client Blogo is amazing. Full-screen, rich-text editing FTW.

Adding functionality to PCS

2009-05-18T22:32:00.001-04:00

Sent the below to GNN, hopefully it'll make its way into PCS.


def inet_lton(integer):
    return struct.pack(">L",integer)

def inet_ltoa(integer):
    return socket.inet_ntoa(inet_lton(integer))
    
def inet_ntol(byteString):
    return struct.unpack(">L",byteString)[0]

def inet_atol(ipString):
    return inet_ntol(socket.inet_aton(ipString))

Real Coding Begins!

2009-05-18T21:03:00.001-04:00

First Step

Create a class that allows filtering of packets based on the IP address and/or port, and works with both IPv4/IPv6 and TCP/TCPv6. Since this is a TCP testing framework, this should allow a lot of isinstance(packet,pcs.packets.tcp.tcp) calls.

Worthwhile Note

2009-05-14T22:50:00.001-04:00

I thought it pertinent to note that I resolved the earlier-mentioned problem by setting the data member in the Packet class directly:


class Packet:
  ...
  data = None
  ...

More PCS Issues, Bugreport

2009-05-13T21:18:00.001-04:00

Issued a bug report on the SF.net tracker. Didn't realized that I wasn't logged in, and subsequently cannot modify the report. Oops.

Anyways, the problem is that dns.py defines the dnsheader class doesn't define the 'data' member. This is present in the Hg revision, which is why I was trying to get it working earlier.

Looks like merging won't be easy, since the newer version relies on is_tcp. Might just add a 'Data' field that sets data=None.

I am beginning to wonder why many of these variables are not defined in the Packet class body, since they are expected to exist. This would also facilitate contextual help systems (like the Python/Eclipse stuff).

Also not sure why is_tcp is used instead of isinstance(self,tcp). Looking back through the code is painful, because I haven't (to this point) found any high-level design documents. Might have to email GNN about this.

Attempted to update to PCS from Hg

2009-05-13T17:15:00.001-04:00

Attempted to update to PCS from Mercurial. Greeted with a bunch of this:


/Users/zach/Documents/workspace/PCS/pcs/bpf/bpf.pyx:391:53: Attempting to index non-array type ''

Error converting Pyrex file to C:
------------------------------------------------------------
...
        li = []
        n = self.bp.bf_len
        if n > 0:
            ip = self.bp.bf_insns
            for 0 <= i < n:
                li.append(op(ip[0].code, ip[0].jt, ip[0].jf, ip[0].k))
                                                              ^
------------------------------------------------------------

Finally!

2009-05-12T22:34:00.001-04:00

Finally, some time to do some investigative work with PCS.

Well, turns out that the code was doing exactly what it should have been doing, just not what I expected.

Consider the following:

sniffer = pcs.PcapConnector(options.interface)

rawPacket = sniffer.read()
etherPacket = ethernet(rawPacket)
     
print "RAW"
print ByteToHex(rawPacket)
print "Ether.bytes"
print ByteToHex(etherPacket.getbytes())
print "Ether.chain().bytes"
print ByteToHex(etherPacket.chain().bytes)

This results in the output:


RAW
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00 45 00 00 4A DD AA 40 00 40 06 54 A1 AC 10 00 0A 40 0C 1C 3C C8 5B 14 46 03 BC CB 8B 8B 23 09 EF 50 18 FF FF EE FD 00 00 2A 02 01 C4 00 1F 00 04 00 14 00 00 00 00 01 C0 00 00 00 00 00 00 00 00 00 01 08 6C 7A 61 62 61 64 61
Ether.bytes
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00
Ether.chain().bytes
00 21 29 A5 A9 3F 00 1B 63 06 82 B2 08 00 45 00 00 4A DD AA 40 00 40 06 54 A1 AC 10 00 0A 40 0C 1C 3C C8 5B 14 46 03 BC CB 8B 8B 23 09 EF 50 18 FF FF EE FD 00 00 2A 02 01 C4 00 1F 00 04 00 14 00 00 00 00 01 C0 00 00 00 00 00 00 00 00 00 01 08 6C 7A 61 62 61 64 61

Looks like the ".getBytes()" function only returns the bytes for that level (which is obviously useful with the Chain object).

Back Home

2009-05-11T17:28:00.001-04:00

Back at home for now, moving to Lansing tomorrow afternoon. Things should really get moving :-D

Turns out that all of my Summer courses are for the first half of the summer. This may result in a disproportionate amount of time being spent after July 2, versus the time spent before it. I still plan to stick fully to all of my previously-outlined commitments, especially this project.

Course Schedule:
HTML
iCal

Embedded:

Moving Back Home

2009-05-09T00:47:00.001-04:00

I'm moving back home from the DC area on Sunday. I've got some of my stuff packed up in the car already, and depending on how tomorrow goes (packing more stuff, tying up loose ends, mailing off eBay packages, biking if there's time) I should get some more time to play with PCS.

Once I get home, I have some errands to run on Monday, and will be moving into my new place in East Lansing on Tuesday. That should give me the rest of the week to get situated, and spend some real time in PCS before the actual "coding period" begins.

Here's to hoping that GNN gets the new version pushed out soon! Looking forward to it :-D

Issues with PCS

2009-05-05T19:43:00.001-04:00

Not sure why the below code doesn't print out the same values three times...


import pcs
from pcs.packets.ethernet import *
from pcs.packets.ipv4 import *
from pcs.packets.tcp import *

def main():
    from optparse import OptionParser

    parser = OptionParser()
    parser.add_option("-i", "--interface",
                      dest="interface", default=None,
                      help="Which interface to use for testing.")
    (options, args) = parser.parse_args()
    sniffer = pcs.PcapConnector(options.interface)
    
    while 1:
        rawPacket = sniffer.read()
        etherPacket = ethernet(rawPacket)
        
        if type(etherPacket.data) == ipv4:
            ipv4Packet = ipv4(etherPacket.data.getbytes())
            
            if type(ipv4Packet.data) == tcp:
                tcpPacket = tcp(ipv4Packet.data.getbytes())
                
                print etherPacket.data.data
                print ipv4Packet.data
                print tcpPacket
        print '-------------------------------------'

Playing around with PCS

2009-05-04T22:56:00.001-04:00

Doing some non-constructive playing around with PCS. I've decided that I may very well use Eclipse with some Python plugins for development, as I've gotten used to the IDE, along with contextual help (member, method lists, etc.) and its debugger.

I'm very, very sorry TextMate. You've served me well.

Installing PCS and Pyrex on a Mac

2009-05-01T22:55:00.001-04:00

Found this out a while ago, but figure it might be useful to users of PCS... (since I couldn't find this information anywhere else)

Download PCS from SF.net
Download Pyrex from canterbury.ac.nz
Extract Pyrex. Then run
```
sudo python setup.py install
```
Extract PCS. Then run
```
sudo python setup.py install
```
Don't bother with the MacPort of pylibpcap. Just download it from SF.net, and run
```
sudo python setup.py install
```
To make sure everything works, bring up Python and type
```
import pcs
```
As long as you don't get any errors, it's safe to assume everything went well :-)

In retrospect, this all is extremely straightforward. I was trying to do it the "MacPorts" way instead of just *doing* it. Would've saved myself a lot of hassle...

Some Time to Work...

2009-05-01T20:35:00.001-04:00

Hopefully I'll have some time to get some things done this weekend, now that I don't have to worry about getting the apartment cleaned up, things sold, and paying for school.

Started getting things set up on my Mac for the project. So far I've got FreeBSD installed in a VMWare machine, Python 2.5 installed (because the MacPort of 2.6 doesn't want to, for whatever reason). Downloaded PCS and started getting it set up, although I've run into a snag getting it to find Pyrex properly.

GSoC 2009 Acceptance

2009-04-20T21:41:00.000-04:00

Google just posted the list of approved student applications, and it looks like mine made it in :-). Hopefully I'll be able to find a blogging client this summer that [1] makes readable HTML and [2] also has decent support for inline code.

Thanks to FreeBSD for being my mentoring organization, and George Neville-Neil for being my mentor. Advance thanks to Prof. Titus Brown for all the hassle I'm about to give him this summer. [/sarcasm?]

More news as it happens.