Saturday, August 29, 2009

Docs Added

Lots of docs added to some of the core components. I'm on my way to the Apple store to have them fix this piece-o-junk, so hopefully I'll have it back soon!


Monday, August 17, 2009

Committed all changes

All changes have been committed to P4 (TcpRegression) and Git (PyPcap). Eveyrthing works on my dev box, but we all know that means nothing. I'll sort out any odds and ends tomorrow. I need to get up for work in 3 hours.

- Zach


Sunday, August 16, 2009

And so it ends

SoC is coming to an end tonight.

Overall, I think the project went very well. I'm going to try to bust out a few deliverable tests tonight since those were the original goal, but I think the volume of work that I got done will speak for itself.

  • I forked PyPcap, fixed bugs, extended the functionality, wrote better tests.
  • I forked PCS-0.5, fixed bugs, extended the functionality (although I didn't write tests for PCS).
  • I've essentially implemented a reference userland implementation of TCP on top of PCS and PyPcap, which should allow for even more applications than a simple regression testing suite, complete with tests of the code itself, as well as a few deliverable tests that should validate functionality of various TCP stacks.

In addition to continuing work with the TcpRegression suite, I've sent an email to the Metasploit project to see if they could use another leisure-pace developer. That should be an interesting project, and maybe I'll be able to apply some of the knowledge about TCP that I gained over the summer. Overall, it's been very productive - for my personal benefit (fiscal and intellectual), and hopefully the FOSS community and FreeBSD as well.

I've got a few minor changes to PyPcap that need to go up -- mostly it's just the inclusion of a function that will print out a string of bytes in the same format that they appear in Wireshark's Packet Bytes view.

There's a load of TcpRegression functionality and tests that will be in the next commit. Over the next week or so, I'll work on the documentation and cleaning up the code, and getting Google their code sample. I'd like a nice, solid "0.1" release. It's also crossed my mind to separate the regression tests themselves from the main framework, and re-badging it "TCPython". The name looks like it hasn't been taken yet, but we'll see how that goes.

Special thanks to Titus Brown and George Neville-Neil for helping me throughout the summer, you both helped me out of a few ruts along the way that could've made the whole project a lot less enjoyable.


Published changes...

Published changes to the PyPcap library. Get the new hotness here:
git clone git://

This build tested on Mac OS X 10.5.7 and FreeBSD 7.1


- included changes from George Neville-Neil's PCS (
- added more tests
- added logging functionality
- implemented patches provided by several users from the original PyPcap site on Google Code. Thanks to...

- Changed some interfaces, but made sure to allow backwards compatibility.

__init__ now has separate args for interface, filename, and dump file.
dump_close from PCS is now closeDumps
the filter can be set by accessing the '.filter' property
each pcap object can now be re-opened after closing with

- openLive
- openOffline
- openDump

- exposed findalldevs() [getxsick]
- exposed lookupnet() [getxsick]
- exposed 'cnt' arg of loop() [getxsick]
- fixed setnonblock, loop() []


Busy Day

Busy day. Added some more PyPcap functionality, fixed some bugs, and implemented tests. All of the functionality is at least touched by the tests...
- Live capture
- Dumping to file
- Reading from file
- Packet filter

Additionally, migrated the 'tcpfilter' class to use bpf instead of manually inspecting the fields of each packet after it had been constructed by PCS. This should be MUCH faster.

Also learned that in Python...

class X():
....y = someObject()

... means that all X objects will be instantiated with the *same instance* of someObject, that is instantiated when the class is declared, instead of each object being allocated when the class instance is created. Weird. I thought it was run when the class was instantiated, and just allowed for the constructor to be a little bit less cluttered. Guess I was wrong (or that there's a Python bug).


Thursday, August 13, 2009

C'mon Nose, stop being a stupid WHAAAAAAAAAT

In python...

>>> t = TestPcap('')
>>> t.testEnumerateInterfaces()
>>> t.testOpenDefaultInterface()
>>> t.testOpenSpecificInterface()


ERROR: testEnumerateInterfaces (testPcap.TestPcap)
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/", line 33, in testEnumerateInterfaces
listOfIfs = pcap.findalldevs()
AttributeError: 'module' object has no attribute 'findalldevs'

ERROR: testOpenSpecificInterface (testPcap.TestPcap)
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/", line 38, in testOpenSpecificInterface
iface = pcap.findalldevs()[0]
AttributeError: 'module' object has no attribute 'findalldevs'

Ran 3 tests in 0.004s

FAILED (errors=2)

Here's the output of a "print dir(pcap)" from within the test, run through nosetests.

-------------------- >> begin captured stdout << ---------------------
['DLT_ARCNET', 'DLT_AX25', 'DLT_CHAOS', 'DLT_EN10MB', 'DLT_EN3MB', 'DLT_FDDI', 'DLT_IEEE802', 'DLT_LINUX_SLL', 'DLT_LOOP', 'DLT_NULL', 'DLT_PFLOG', 'DLT_PFSYNC', 'DLT_PPP', 'DLT_PRONET', 'DLT_RAW', 'DLT_SLIP', '__author__', '__builtins__', '__copyright__', '__doc__', '__file__', '__license__', '__maintainer__', '__name__', '__revison__', '__url__', '__version__', 'bpf', 'calendar', 'dltoff', 'ex_name', 'lookupdev', 'pcap', 'sys', 'time']
--------------------- >> end captured stdout << ----------------------

Obviously, it's not there. Pcap is imported as:

import pcs.pcap as pcap
import pcs

Here's the output of "print pcs.__file__" from Python:

Aaaand from Nose-tests:




Lost track of time tonight, stayed up way later than I wanted to. Overhauled a LOT of code in the PyPcap library (both C any Python code). It's now a lot cleaner, and a lot of the functionality that was only available via creating a new pcap object is now available at runtime (i.e. open/reopen an interface or offline capture).

Also changed a bit of the Windows code that (for whatever reason) attempted to implement pcap_lookupdev, even though WinPcap has that function. Other minor changes were included in the C code (a little bit of refactoring).

The Python code now includes logging for most of its functionality, which is disabled by default. It does require either [1] modifying the PYX file or [2] modifying pcap.DEBUG_LEVEL before instantiating the pcap object that logging is desired for. Lots of refactoring. Hopefully I didn't break anything.

Must. Get. To. Sleeeeeeeeeeep.


Wednesday, August 12, 2009

Nope, it was Python.

The __next__ method

Extension types wishing to implement the iterator interface should define a method called __next__, not next. The Python system will automatically supply a next method which calls your__next__. Do NOT explicitly give your type a next method, or bad things could happen.



Changing the name of "__next__"... what the hell, Python? (Although I'm much more apt to blame Pyrex)

File "<stdin>", line 1, in <module>

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/", line 66, in read
return self.pcapHandle.readpkt()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/", line 1009, in readpkt
packet =

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/", line 991, in read
packet =



Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/", line 66, in read
return self.pcapHandle.readpkt()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/", line 1009, in readpkt
packet =
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/", line 991, in read
packet =
File "pcap.pyx", line 502, in pcap.pcap.__next__


Not understanding the logic flow here

Going back and testing stuff. Check this out... #

which calls:

return self.pcapHandle.readpkt() # pcs.PcapConnector.readpkt()

which calls:

packet =[1] #

However, debug statements show

"IN __NEXT__".

Dubbleyou Tee Eff.



Finally figured out the logging bit... a few changes to the config file, combined with:

logFiles  = [ 'logging.conf',
join( expanduser( '~' ), 'logging.conf' ),
join( dirname( __file__ ), 'logging.conf' ),
None ]

for logFile in logFiles:
if exists( logFile ):
print "Using logfile %s" % logFile
logging.config.fileConfig( logFile )

Fixed the problem


Monday, August 10, 2009

Modem dead

Posting from iPhone. My router & modem fried in storm. :-/ fml Read more...

The Issue [Nevermind, Fixed]

Nevermind, in the process of writing this post, I made a change at some point that made everything work all honky-dory. in pcs-0.5 and pypcap both use the following command to link everything:

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap_ex.o -L/usr/lib -lpcap -o build/lib.macosx-10.5-i386-2.6/pcs/ in tcpregression links it with...

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap_ex.o -o build/lib.macosx-10.5-i386-2.6/pcs/

The discrepancy between the two being:

-L/usr/lib -lpcap

Exactly why this is the case, I'm unsure. Obviously, if isn't linked with libpcap (the actual pcap library), there would be issues.

The build system uses from pcs-0.5, with a modification to include the different source directory. Here's the diff between pcs' and my "". All of the changes have to do with the source being in a "src" folder.

< pcap_cache = 'src/pcs/pcap/config.pkl'
> pcap_cache = 'pcs/pcap/config.pkl'
< f = open( 'src/pcs/pcap/config.h', 'w' )
> f = open( 'pcs/pcap/config.h', 'w' )
< sources = [ 'src/pcs/pcap/pcap.pyx', 'src/pcs/pcap/pcap_ex.c' ],
> sources = [ 'pcs/pcap/pcap.pyx', 'pcs/pcap/pcap_ex.c' ],
> setup( name = 'pcs',
> version = '0.5',
> description = 'Packet Construction Set',
> author = 'George V. Neville-Neil',
> author_email = '',
> url = '',
> packages = ['pcs', 'pcs.packets'],
> cmdclass = pcap_cmds,
> ext_modules = [ pcap ],
> )

And then the actual setup file....

from distutils.core import setup, Extension
import pcssetup
setup( name = 'tcpregression',
version = '1.0',
description = 'FreeBSD TCP Regression Suite',
author = 'Zach Riggle',
author_email = '',
url = '',
packages = ['tcpregression',
package_dir = {'':'src'},
cmdclass = pcssetup.pcap_cmds,
ext_modules = [pcssetup.pcap],

As you can see, not too many changes.


Proper Building with

Thought I'd give proper building with "" a go. Back to square one with:

ImportError: dlopen(pcs/, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/build/lib.macosx-10.5-i386-2.6/pcs/
Expected in: dynamic lookup



Massive Cleanup

I'm going back through a lot of the code, removing commented-out code, deleting un-used files, and restructuring a bit. Hopefully this'll make everything a lot cleaner for when I'm done :-)


Sunday, August 9, 2009

Problem Solved

Evidently " build_ext -i" doesn't do somethign that the regular build operation does. Not sure what. Anyway, the Python documentation for says that the output files will always end up in build/lib/ or build/lib.arch/ so a "cp build/lib*/ ..." will take care of it. Hopefully that'll solve the problem.


PyPcap issues when running in-directory

Some of the issues I originally had when getting PyPcap setup back in June are re-surfacing, now that I've got PCS and PyPcap (as pcs.pcap) inside of the tcpregression folder:

>>> import pcs.pcap
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pcs/", line 68, in <module>
import pcs.pcap as pcap
ImportError: dlopen(pcs/, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/src/tcpregression/pcs/
Expected in: dynamic lookup


Saturday, August 8, 2009

Build system

I've integrated the PyPcap changes ( into the TcpRegression Perforce tree, and put the custom-modified PCS-0.5 directory in there, as well. It contains some bugfixes for PCS-0.5, as well as a few modifications to make things easier for my particular uses.

The build system for building PyPcap, and putting the module into the PCS directory is also in place. That was kind of an interesting foray into make and, but it works very cleanly now that I've got it figured out (thanks Titus!).

Next steps are to write a few quick tests for the TcpRegression library itself, mainly to make sure that the threading stuff works as expected. Assuming that it does, I'll be able to crank out some TCP tests and have a final product for FreeBSD.

My work schedule changes quite a bit next week, so I'll be awake while most people are asleep/at work (depending on how I do it). Basically I'm working 0200-1000 next week, and need to figure out if I'm gonna wake just before 2AM, or go to bed just after 10AM.


Thursday, August 6, 2009

GitHub site, issues

"git push".

Now the stuff is ACTUALLY on the site. My god is the default color scheme ugly. Working to fix that!

If anybody is familiar with, please contact me via email.


Wednesday, August 5, 2009

PyPcap Migrated to Git

A bunch of changes have been made. I'm working on going back in and documenting those changes. I'll have a changelog shortly, as well as a demo.


Forked, Fixed, Updating

Taking some advice and forking PyPcap. Working on adding the patches provided by users. Right now I'm trying to get Pyrex to like the changes that I'm making to make the 'findalldevs' function closer to the real one. A user submitted a patch on the Google Code site, but it doesn't return ALL of the data the function should. Unfortunately, I have to add a bunch of types and unroll several linked-lists and arrays to get all of the data.


Sunday, August 2, 2009

Perforce Woes

I don't know if I mentioned this earlier, but Perforce is *by far* the hardest version management system that I've ever had to work with -- out of CVS and SVN, that is.

First off, it defines its operation off of environment variables. This makes manipulation pretty hard unless I define environment variables in bashrc or something -- and changing environments pretty hard to boot.
Second off, it complains every time my hostname changes. Right now, it keeps complaining that I can only commit from "Zachs-Computer", which is the hostname.
Third, I have to define a workspace. This in itself is a pain. I can see the benefits, although if I have to change my hostname to use a workspace, it's kind of moot point. "svn co path/to/server" FTW.

I'm back off to fight with P4 while I try to commit my code :-\


PyPcap = Abandonware

So PyPcap, the Pcap library that PCS includes in its distribution, is essentially abandonware. Hasn't been a new release since Jan 2007. The files in the included PyPcap say that GNN is the maintainer.

George, if you are reading this, please let me know so that I may submit patches to PyPcap. There is at least one issue that is of high importance to my project, for which a patch already exists, that I don't mind integrating into the PyPcap project on Google Code.

I tweeted at @dugsong, so hopefully the guy in charge of the project (as far as Google Code is concerned) will give me access to make the changes. I can make the changes locally, but there's no reason not to include them online for all to use.


Saturday, August 1, 2009

Okay, so that's done... but we're stuck with a hanging thread [HALP!]

The full TCP exchange of 3-way-handshake, data transfer (with acknowledgement) and active-close (Established -> Fin Wait 1 -> Fin Wait 2 -> Time Wait) has been successfully tested, and works fine.

Here's the script, and subsequent output:

import testconfig
from tcpstatemachine import *
from threading import Thread
from testserver import runServer
from multiprocessing import Process

def test():
t = TcpStateMachine()
t.die = False

serverProcess = Process(target=runServer)


>>> import test
2009-08-01 17:35:18,212 - tcpFilter - INFO - Opened lo0
>>> test.test()
2009-08-01 17:35:19,867 - TcpStateMachine - DEBUG - __init__ - (('', 31765), ('', 32261))
2009-08-01 17:35:19,870 - tcpFilter - DEBUG - openInterface - Tried to re-open same interface: lo0
2009-08-01 17:35:19,870 - tcpFilter - INFO - Opened lo0
2009-08-01 17:35:19,870 - TcpStateMachine - INFO - Local host:
2009-08-01 17:35:19,871 - TcpStateMachine - INFO - Remote host:
~~~~~ In runServer
~~~~~ Listening on
~~~~~ Waiting for accepted connection
2009-08-01 17:35:19,950 - TcpStateMachine - DEBUG - open - In valid state to open connection
2009-08-01 17:35:19,967 - TcpStateMachine - DEBUG - open - Generated ISS: 1036344287
2009-08-01 17:35:19,969 - TcpStateMachine - DEBUG - open - Sending SYN packet
2009-08-01 17:35:19,975 - TcpStateMachine - STATE - Setting state from Closed to Syn-Sent
2009-08-01 17:35:19,976 - TcpRecvDaemon - INFO - Starting receive thread for object TcpStateMachine( (,31765), (,32261) )
2009-08-01 17:35:19,977 - TcpSendDaemon - INFO - Starting send thread for object TcpStateMachine( (,31765), (,32261) )
2009-08-01 17:35:19,980 - TcpStateMachine - DEBUG - open - Waiting for state == ESTABLISHED... Outbound Sequenes: segmentBuffer(limit=4294967296,base=1036344287,copyList=['[SYN]']+[])
2009-08-01 17:35:20,047 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344287 to 1036345633 in ['[SYN]']
2009-08-01 17:35:20,049 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344287 to 1036344288: ['[SYN]']
2009-08-01 17:35:20,050 - TcpSendDaemon - DEBUG - sendThread - Haz a SYN!
2009-08-01 17:35:20,054 - TcpStateMachine - SENT - [<class tcp Seq: 1036344287, 31765->32261, [S], reserved: 0, checksum: 7711, offset: 5, window: 65535L>,
<class payload {payload: ''}>]
2009-08-01 17:35:20,484 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:20,485 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201111, 32261->31765, [AS], reserved: 0, checksum: 65058, offset: 6, window: 65535, ack_number: 1036344288>]
2009-08-01 17:35:20,487 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] IN SYN_SENT
2009-08-01 17:35:20,489 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Has ACK
2009-08-01 17:35:20,490 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Accepted ACK
2009-08-01 17:35:20,492 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Has SYN and either good ACK or no ACK
2009-08-01 17:35:20,492 - TcpStateMachine - STATE - Setting state from Syn-Sent to Established
Sending ack: {seq: 1036344288, ack: 1, ack#: 1883201112}
2009-08-01 17:35:20,497 - TcpStateMachine - SENT - [<class tcp Seq: 1036344288, 31765->32261, [A], reserved: 0, checksum: 20344, offset: 5, window: 65535L, ack_number: 1883201112L>]
~~~~~ Connected
~~~~~ Waiting on Data
2009-08-01 17:35:20,499 - TcpStateMachine - DEBUG - open - Established
2009-08-01 17:35:20,507 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344288 to 1036345634 in ['C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!', 'C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!']
2009-08-01 17:35:20,508 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344288 to 1036344308: ['C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!', 'C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!']
2009-08-01 17:35:20,511 - TcpStateMachine - SENT - [<class tcp Seq: 1036344288, 31765->32261, [A], reserved: 0, checksum: 21671, offset: 5, window: 65535L, ack_number: 1883201112L>,
<class payload {payload: 'COOOOOOOL!COOOOOOOL!'}>]
~~~~~ Waiting on Data
2009-08-01 17:35:21,008 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:21,009 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [A], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344308>]
2009-08-01 17:35:21,009 - TcpStateMachine - ERROR - This point should never be reached!
2009-08-01 17:35:21,059 - TcpStateMachine - STATE - Setting state from Established to Fin-Wait-1
2009-08-01 17:35:22,021 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344308 to 1036345654 in ['[FIN]']
2009-08-01 17:35:22,023 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344308 to 1036344309: ['[FIN]']
2009-08-01 17:35:22,024 - TcpSendDaemon - DEBUG - sendThread - Haz an FIN!
2009-08-01 17:35:22,028 - TcpStateMachine - SENT - [<class tcp Seq: 1036344308, 31765->32261, [AF], reserved: 0, checksum: 20323, offset: 5, window: 65535L, ack_number: 1883201112L>,
<class payload {payload: ''}>]
Received Data:
~~~~~ No more data
2009-08-01 17:35:23,016 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:23,017 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [A], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344309>]
2009-08-01 17:35:23,019 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-1] Our FIN is ACKed
2009-08-01 17:35:23,019 - TcpStateMachine - STATE - Setting state from Fin-Wait-1 to Fin-Wait-2
2009-08-01 17:35:23,020 - TcpStateMachine - ERROR - This point should never be reached!
2009-08-01 17:35:23,023 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:23,024 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [AF], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344309>]
2009-08-01 17:35:23,025 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-2] connection closing
2009-08-01 17:35:23,027 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-2] sending ACK of FIN
Sending ack: {seq: 1036344309, ack: 1, ack#: 1883201113}
2009-08-01 17:35:23,029 - TcpStateMachine - SENT - [<class tcp Seq: 1036344309, 31765->32261, [A], reserved: 0, checksum: 20322, offset: 5, window: 65535L, ack_number: 1883201113L>]
2009-08-01 17:35:23,030 - TcpStateMachine - STATE - Setting state from Fin-Wait-2 to Time-Wait
2009-08-01 17:35:23,031 - TcpStateMachine - DEBUG - debugWithState - [Time-Wait] Starting TIME_WAIT timer
2009-08-01 17:35:24,035 - TcpSendDaemon - DEBUG - sendThread - Quitting send thread
2009-08-01 17:35:33,032 - TcpStateMachine - STATE - Setting state from Time-Wait to Closed

It's a bit more verbose than it needs to be, and there's a few things that will be taken out (i.e. "Haz a SYN!") but overall I'm pleased with the way it works.

However, the only remaining problem before the real-real test-writing gets going is that the call to retrieve the next packet from PCAP is a blocking call that's performed by a Python thread (which has no terminate() or equivalent method).

Looking into the issue a bit, pypcap has a "setnonblock(nonblock=True)" method, but due to the way everything is done, this isn't quite so easy to work with (there are also some bug reports that say that this functionality doesn't necessarily work as expected).