Holy Crap How Time Flies

Lots of time has gone by since I've had a chance to put some hours into the TCP Regression framework. I haven't forgotten it, just been busy. A trip to New York City, followed by Halloween, and two absolutely gorgeous November sundays kept me away from the computer for the past few weeks.

The coming weekends, since I'm not going home for Thanksgiving, as well as Christmas-time (since most of the other interns leave Dec. 17, and I am staying until Dec. 31) will probably see a spark of productivity.

Read more...

Build Issues when Installing

In the process of investigating the build issues on Mac OS X 10.6 Snow Leopard, I decided to build on FreeBSD.

The combo:

python setup.py config

python setup.py build

sudo python setup.py test

Works just fine. However, once the package is installed... things go haywire (full output below). The issue appears to be in the final linking stage.

make

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

make install

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

For whatever reason, libpcap isn't being linked in. The tests attempt to import pcap, then make sure that it is valid, and if that doesn't work, import pcs.pcap (as seen at the end). When attempting to import the installed version of pcap, the "import pcap" statement fails. Grrrr...

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ ls

CHANGES LICENSE Makefile README pcap.c pcap.pyx pcap_ex.c pcap_ex.h setup.py tests testsniff.py

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ make

pyrexc pcap.pyx

python setup.py config

running config

found {'libraries': ['pcap'], 'library_dirs': ['/usr/lib'], 'include_dirs': ['/usr/include']}

python setup.py build

running build

running build_ext

pyrexc pcap.pyx --> pcap.c

building 'pcap' extension

creating build

creating build/temp.freebsd-7.1-RELEASE-i386-2.5

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o

pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':

pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':

pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_dump':

pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o

pcap_ex.c: In function 'pcap_ex_next':

pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type

creating build/lib.freebsd-7.1-RELEASE-i386-2.5

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test

python setup.py test

running test

testDispatch (tests.testPcap.TestPcap) ... ok

testEnumerateInterfaces (tests.testPcap.TestPcap) ... ok

testErrors (tests.testPcap.TestPcap) ... ok

testInjectedPacketIsReceived (tests.testPcap.TestPcap) ... ok

testIter (tests.testPcap.TestPcap) ... ok

testIterable (tests.testPcap.TestPcap) ... ok

testOpenDefaultInterface (tests.testPcap.TestPcap) ... ok

testOpenLive (tests.testPcap.TestPcap) ... ok

testPacketFilter (tests.testPcap.TestPcap) ... ok

testProperties (tests.testPcap.TestPcap) ... ok

testReadpkts (tests.testPcap.TestPcap) ... ok

testWriteDump_ReadOffline (tests.testPcap.TestPcap) ... ok

----------------------------------------------------------------------

Ran 12 tests in 1.541s

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make install

python setup.py install

running install

running build

running build_ext

pyrexc pcap.pyx --> pcap.c

building 'pcap' extension

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o

pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':

pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':

pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_dump':

pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o

pcap_ex.c: In function 'pcap_ex_next':

pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

running install_lib

copying build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so -> /usr/local/lib/python2.5/site-packages

running install_egg_info

Removing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info

Writing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test

python setup.py test

running test

Traceback (most recent call last):

File "setup.py", line 120, in

provides = ['pcap'] )

File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup

dist.run_commands()

File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands

self.run_command(cmd)

File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command

cmd_obj.run()

File "setup.py", line 91, in run

from tests.testPcap import PcapTestSuite

File "/usr/home/zjriggl/zachriggle-pypcap-1f0484c/tests/testPcap.py", line 17, in

import pcs.pcap as pcap

ImportError: No module named pcs.pcap

*** Error code 1

Stop in /usr/home/zjriggl/zachriggle-pypcap-1f0484c.

Read more...

PCS-0.5 Diff with PCS version used in TcpRegression

I've created a Diff file that shows all of the changes that I made to PCS over the course of the summer. Some of these might be merged back into the PCS mainline, others might not.

The highlight of the changes are:

• BitString class. The code for encoding individual bits, even if there is a lack of alignment on 8-bit boundaries, is a bit iffy (even the code says "This makes my head hurt"). The BitString class allows manipulation of individual bits just like a Python list object, and can be translated into an integer value, string value, binary string (via bin()), hex string (via hex()), and allows arbitrary manipulation.
• The names of the pcs.packets.tcp.tcp class fields were assigned variables. Instead of object['fieldName'] which is prone to typos going undetected, I made a bunch of variables so that they can be referred to as object[f_fieldname]. If mistyped, the interpreter picks up on the nonexistent variable.

GNN: I've explicitly emailed you about this, as this was something that you requested.

Everyone else can get a copy here:

http://p4db.freebsd.org/fileDownLoad.cgi?FSPC=//depot/projects/soc2009/zjriggl%5ftcpregression/diff/pcs%2d0.5.diff&REV=1

From looking at the diff, it does not appear that my changes rely on anything in the forked PCap library (and for that matter, don't involve PCap at all).

Zach

Read more...

Yep, 10.6 Did It (Methinks)

The build.py script was only set to look in the first include directory for pcap.h. On 10.6, /usr/include/pcap.h includes /usr/local/pcap/pcap.h, which contains all of the definitions the build system was looking for.

Fixed the build script, will add it to Git soon. Read more...

10.6 Breaks PyPcap?

I think Snow Leopard breaks PyPcap -- for whatever reason, I'm getting compile errors that didn't previously exist. I'll have to boot up one of the VMs and see if everything still works OK, or if there was some change that I didn't properly test that I also completely forgot that I made in the first place. Read more...

Docs Added

Lots of docs added to some of the core components. I'm on my way to the Apple store to have them fix this piece-o-junk, so hopefully I'll have it back soon!

Read more...

Committed all changes

All changes have been committed to P4 (TcpRegression) and Git (PyPcap). Eveyrthing works on my dev box, but we all know that means nothing. I'll sort out any odds and ends tomorrow. I need to get up for work in 3 hours.

- Zach

Read more...

And so it ends

SoC is coming to an end tonight.

Overall, I think the project went very well. I'm going to try to bust out a few deliverable tests tonight since those were the original goal, but I think the volume of work that I got done will speak for itself.

• I forked PyPcap, fixed bugs, extended the functionality, wrote better tests.
  
• I forked PCS-0.5, fixed bugs, extended the functionality (although I didn't write tests for PCS).
  
• I've essentially implemented a reference userland implementation of TCP on top of PCS and PyPcap, which should allow for even more applications than a simple regression testing suite, complete with tests of the code itself, as well as a few deliverable tests that should validate functionality of various TCP stacks.
  

In addition to continuing work with the TcpRegression suite, I've sent an email to the Metasploit project to see if they could use another leisure-pace developer. That should be an interesting project, and maybe I'll be able to apply some of the knowledge about TCP that I gained over the summer. Overall, it's been very productive - for my personal benefit (fiscal and intellectual), and hopefully the FOSS community and FreeBSD as well.

I've got a few minor changes to PyPcap that need to go up -- mostly it's just the inclusion of a function that will print out a string of bytes in the same format that they appear in Wireshark's Packet Bytes view.

There's a load of TcpRegression functionality and tests that will be in the next commit. Over the next week or so, I'll work on the documentation and cleaning up the code, and getting Google their code sample. I'd like a nice, solid "0.1" release. It's also crossed my mind to separate the regression tests themselves from the main framework, and re-badging it "TCPython". The name looks like it hasn't been taken yet, but we'll see how that goes.

Special thanks to Titus Brown and George Neville-Neil for helping me throughout the summer, you both helped me out of a few ruts along the way that could've made the whole project a lot less enjoyable.

Read more...

Published changes...

Published changes to the PyPcap library. Get the new hotness here:

http://zachriggle.github.com/pypcap/
or
git clone git://github.com/zachriggle/pypcap

This build tested on Mac OS X 10.5.7 and FreeBSD 7.1

CHANGELOG

pypcap-1.2:
- included changes from George Neville-Neil's PCS (http://pcs.sf.net)
- added more tests
- added logging functionality
- implemented patches provided by several users from the original PyPcap site on Google Code. Thanks to...
getxsick
dirk-loss.de

- Changed some interfaces, but made sure to allow backwards compatibility.
Examples:

__init__ now has separate args for interface, filename, and dump file.
dump_close from PCS is now closeDumps
the filter can be set by accessing the '.filter' property
each pcap object can now be re-opened after closing with

- openLive
- openOffline
- openDump

- exposed findalldevs() [getxsick]
- exposed lookupnet() [getxsick]
- exposed 'cnt' arg of loop() [getxsick]
- fixed setnonblock, loop() [dirk-loss.de]

Read more...

Busy Day

Busy day. Added some more PyPcap functionality, fixed some bugs, and implemented tests. All of the functionality is at least touched by the tests...
- Live capture
- Dumping to file
- Reading from file
- Packet filter

Additionally, migrated the 'tcpfilter' class to use bpf instead of manually inspecting the fields of each packet after it had been constructed by PCS. This should be MUCH faster.

Also learned that in Python...

class X():
....y = someObject()

... means that all X objects will be instantiated with the *same instance* of someObject, that is instantiated when the class is declared, instead of each object being allocated when the class instance is created. Weird. I thought it was run when the class was instantiated, and just allowed for the constructor to be a little bit less cluttered. Guess I was wrong (or that there's a Python bug).

Read more...

C'mon Nose, stop being a stupid WHAAAAAAAAAT

In python...

>>> t = TestPcap('')
>>> t.testEnumerateInterfaces()
>>> t.testOpenDefaultInterface()
>>> t.testOpenSpecificInterface()

But...

zach@Zachs-Computer:~/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests$nosetests testPcap.py
E.E
======================================================================
ERROR: testEnumerateInterfaces (testPcap.TestPcap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 33, in testEnumerateInterfaces
listOfIfs = pcap.findalldevs()
AttributeError: 'module' object has no attribute 'findalldevs'

======================================================================
ERROR: testOpenSpecificInterface (testPcap.TestPcap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 38, in testOpenSpecificInterface
iface = pcap.findalldevs()[0]
AttributeError: 'module' object has no attribute 'findalldevs'

----------------------------------------------------------------------
Ran 3 tests in 0.004s

FAILED (errors=2)

Here's the output of a "print dir(pcap)" from within the test, run through nosetests.

-------------------- >> begin captured stdout << ---------------------
['DLT_ARCNET', 'DLT_AX25', 'DLT_CHAOS', 'DLT_EN10MB', 'DLT_EN3MB', 'DLT_FDDI', 'DLT_IEEE802', 'DLT_LINUX_SLL', 'DLT_LOOP', 'DLT_NULL', 'DLT_PFLOG', 'DLT_PFSYNC', 'DLT_PPP', 'DLT_PRONET', 'DLT_RAW', 'DLT_SLIP', '__author__', '__builtins__', '__copyright__', '__doc__', '__file__', '__license__', '__maintainer__', '__name__', '__revison__', '__url__', '__version__', 'bpf', 'calendar', 'dltoff', 'ex_name', 'lookupdev', 'pcap', 'sys', 'time']
--------------------- >> end captured stdout << ----------------------

Obviously, it's not there. Pcap is imported as:

try:
import pcs.pcap as pcap
except:
import pcs

Here's the output of "print pcs.__file__" from Python:
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/pcap.so

Aaaand from Nose-tests:
/Library/Python/2.5/site-packages/pcs/pcap.so

Whaaaaaaaaaaat?

Read more...

Time

Lost track of time tonight, stayed up way later than I wanted to. Overhauled a LOT of code in the PyPcap library (both C any Python code). It's now a lot cleaner, and a lot of the functionality that was only available via creating a new pcap object is now available at runtime (i.e. open/reopen an interface or offline capture).

Also changed a bit of the Windows code that (for whatever reason) attempted to implement pcap_lookupdev, even though WinPcap has that function. Other minor changes were included in the C code (a little bit of refactoring).

The Python code now includes logging for most of its functionality, which is disabled by default. It does require either [1] modifying the PYX file or [2] modifying pcap.DEBUG_LEVEL before instantiating the pcap object that logging is desired for. Lots of refactoring. Hopefully I didn't break anything.

Must. Get. To. Sleeeeeeeeeeep.

Read more...

Nope, it was Python.

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html

The __next__ method

Extension types wishing to implement the iterator interface should define a method called __next__, not next. The Python system will automatically supply a next method which calls your__next__. Do NOT explicitly give your type a next method, or bad things could happen.

Read more...

Wtf?

Changing the name of "__next__"... what the hell, Python? (Although I'm much more apt to blame Pyrex)

File "<stdin>", line 1, in <module>

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read
return self.pcapHandle.readpkt()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt
packet = self.read()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read
packet = self.file.next()

Read more...

Yep.

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read
return self.pcapHandle.readpkt()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt
packet = self.read()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read
packet = self.file.next()
File "pcap.pyx", line 502, in pcap.pcap.__next__
KeyboardInterrupt

Read more...

Not understanding the logic flow here

Going back and testing stuff. Check this out...

self.filter.read() # tcpregression.tcpfilter.TcpFilter.read()

which calls:

return self.pcapHandle.readpkt() # pcs.PcapConnector.readpkt()

which calls:

packet = self.file.next()[1] # pcs.pcap.pcap.next()

However, debug statements show

"IN __NEXT__".

Dubbleyou Tee Eff.

Read more...

HA!

Finally figured out the logging bit... a few changes to the config file, combined with:

logFiles  = [ 'logging.conf',
             join( expanduser( '~'  ), 'logging.conf'  ), 
             join( dirname( __file__ ), 'logging.conf'  ), 
             None ] 
  
for  logFile in  logFiles: 
     if  exists( logFile ): 
         print  "Using  logfile  %s"  % logFile 
         logging.config.fileConfig( logFile ) 

Fixed the problem

Read more...

Modem dead

Posting from iPhone. My router & modem fried in storm. :-/ fml Read more...

The Issue [Nevermind, Fixed]

Nevermind, in the process of writing this post, I made a change at some point that made everything work all honky-dory.

build.py in pcs-0.5 and pypcap both use the following command to link everything:

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap_ex.o -L/usr/lib -lpcap -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so

build.py in tcpregression links it with...

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap_ex.o -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so

The discrepancy between the two being:

-L/usr/lib -lpcap

Exactly why this is the case, I'm unsure. Obviously, if pcap.so isn't linked with libpcap (the actual pcap library), there would be issues.

The build system uses setup.py from pcs-0.5, with a modification to include the different source directory. Here's the diff between pcs' setup.py and my "pcssetup.py". All of the changes have to do with the source being in a "src" folder.

< pcap_cache = 'src/pcs/pcap/config.pkl'
---
> pcap_cache = 'pcs/pcap/config.pkl'
71c71
<         f = open( 'src/pcs/pcap/config.h', 'w' )
---
>         f = open( 'pcs/pcap/config.h', 'w' )
122c122
<                  sources = [ 'src/pcs/pcap/pcap.pyx', 'src/pcs/pcap/pcap_ex.c' ],
---
>                  sources = [ 'pcs/pcap/pcap.pyx', 'pcs/pcap/pcap_ex.c' ],
129a130,140
> 
> setup( name = 'pcs',
>       version = '0.5',
>       description = 'Packet Construction Set',
>       author = 'George V. Neville-Neil',
>       author_email = 'gnn@neville-neil.com',
>       url = 'http://pcs.sf.net',
>       packages = ['pcs', 'pcs.packets'],
>       cmdclass = pcap_cmds,
>       ext_modules = [ pcap ],
>       )

And then the actual setup file....

from distutils.core import setup, Extension
import pcssetup
setup( name = 'tcpregression',
version = '1.0',
description = 'FreeBSD TCP Regression Suite',
author = 'Zach Riggle',
author_email = 'zjriggl@freebsd.org',
url = 'http://freebsd.org',
packages = ['tcpregression',
'tcpregression.pcsextension',
'tcpregression.tests',
'pcs',
'pcs.packets'],
package_dir = {'':'src'},
cmdclass = pcssetup.pcap_cmds,
ext_modules = [pcssetup.pcap],
)

As you can see, not too many changes.

Read more...

Proper Building with Setup.py

Thought I'd give proper building with "setup.py" a go. Back to square one with:

ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/build/lib.macosx-10.5-i386-2.6/pcs/pcap.so
Expected in: dynamic lookup

Humbug.

Read more...