Monday, November 16, 2009

Holy Crap How Time Flies

Lots of time has gone by since I've had a chance to put some hours into the TCP Regression framework. I haven't forgotten it, just been busy. A trip to New York City, followed by Halloween, and two absolutely gorgeous November sundays kept me away from the computer for the past few weeks.

The coming weekends, since I'm not going home for Thanksgiving, as well as Christmas-time (since most of the other interns leave Dec. 17, and I am staying until Dec. 31) will probably see a spark of productivity.
Read more...

Sunday, October 11, 2009

Build Issues when Installing

In the process of investigating the build issues on Mac OS X 10.6 Snow Leopard, I decided to build on FreeBSD.

The combo:
python setup.py config
python setup.py build
sudo python setup.py test

Works just fine. However, once the package is installed... things go haywire (full output below). The issue appears to be in the final linking stage.

make
cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

make install
cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

For whatever reason, libpcap isn't being linked in. The tests attempt to import pcap, then make sure that it is valid, and if that doesn't work, import pcs.pcap (as seen at the end). When attempting to import the installed version of pcap, the "import pcap" statement fails. Grrrr...

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ ls
CHANGES LICENSE Makefile README pcap.c pcap.pyx pcap_ex.c pcap_ex.h setup.py tests testsniff.py
[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ make
pyrexc pcap.pyx
python setup.py config
running config
found {'libraries': ['pcap'], 'library_dirs': ['/usr/lib'], 'include_dirs': ['/usr/include']}
python setup.py build
running build
running build_ext
pyrexc pcap.pyx --> pcap.c
building 'pcap' extension
creating build
creating build/temp.freebsd-7.1-RELEASE-i386-2.5
cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o
pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':
pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type
pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':
pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type
pcap.c: In function '__pyx_f_4pcap_4pcap_dump':
pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type
cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o
pcap_ex.c: In function 'pcap_ex_next':
pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type
creating build/lib.freebsd-7.1-RELEASE-i386-2.5
cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so
[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test
python setup.py test
running test
testDispatch (tests.testPcap.TestPcap) ... ok
testEnumerateInterfaces (tests.testPcap.TestPcap) ... ok
testErrors (tests.testPcap.TestPcap) ... ok
testInjectedPacketIsReceived (tests.testPcap.TestPcap) ... ok
testIter (tests.testPcap.TestPcap) ... ok
testIterable (tests.testPcap.TestPcap) ... ok
testOpenDefaultInterface (tests.testPcap.TestPcap) ... ok
testOpenLive (tests.testPcap.TestPcap) ... ok
testPacketFilter (tests.testPcap.TestPcap) ... ok
testProperties (tests.testPcap.TestPcap) ... ok
testReadpkts (tests.testPcap.TestPcap) ... ok
testWriteDump_ReadOffline (tests.testPcap.TestPcap) ... ok

----------------------------------------------------------------------
Ran 12 tests in 1.541s
[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make install
python setup.py install
running install
running build
running build_ext
pyrexc pcap.pyx --> pcap.c
building 'pcap' extension
cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o
pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':
pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type
pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':
pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type
pcap.c: In function '__pyx_f_4pcap_4pcap_dump':
pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type
cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o
pcap_ex.c: In function 'pcap_ex_next':
pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type
cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so
running install_lib
copying build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so -> /usr/local/lib/python2.5/site-packages
running install_egg_info
Removing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info
Writing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info
[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test
python setup.py test
running test
Traceback (most recent call last):
File "setup.py", line 120, in
provides = ['pcap'] )
File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup
dist.run_commands()
File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands
self.run_command(cmd)
File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
cmd_obj.run()
File "setup.py", line 91, in run
from tests.testPcap import PcapTestSuite
File "/usr/home/zjriggl/zachriggle-pypcap-1f0484c/tests/testPcap.py", line 17, in
import pcs.pcap as pcap
ImportError: No module named pcs.pcap
*** Error code 1

Stop in /usr/home/zjriggl/zachriggle-pypcap-1f0484c.



Read more...

PCS-0.5 Diff with PCS version used in TcpRegression

I've created a Diff file that shows all of the changes that I made to PCS over the course of the summer. Some of these might be merged back into the PCS mainline, others might not.

The highlight of the changes are:
  • BitString class. The code for encoding individual bits, even if there is a lack of alignment on 8-bit boundaries, is a bit iffy (even the code says "This makes my head hurt"). The BitString class allows manipulation of individual bits just like a Python list object, and can be translated into an integer value, string value, binary string (via bin()), hex string (via hex()), and allows arbitrary manipulation.
  • The names of the pcs.packets.tcp.tcp class fields were assigned variables. Instead of object['fieldName'] which is prone to typos going undetected, I made a bunch of variables so that they can be referred to as object[f_fieldname]. If mistyped, the interpreter picks up on the nonexistent variable.
GNN: I've explicitly emailed you about this, as this was something that you requested.

Everyone else can get a copy here:

From looking at the diff, it does not appear that my changes rely on anything in the forked PCap library (and for that matter, don't involve PCap at all).

Zach
Read more...

Sunday, September 20, 2009

Yep, 10.6 Did It (Methinks)

The build.py script was only set to look in the first include directory for pcap.h. On 10.6, /usr/include/pcap.h includes /usr/local/pcap/pcap.h, which contains all of the definitions the build system was looking for.

Fixed the build script, will add it to Git soon. Read more...

Sunday, September 13, 2009

10.6 Breaks PyPcap?

I think Snow Leopard breaks PyPcap -- for whatever reason, I'm getting compile errors that didn't previously exist. I'll have to boot up one of the VMs and see if everything still works OK, or if there was some change that I didn't properly test that I also completely forgot that I made in the first place. Read more...

Saturday, August 29, 2009

Docs Added

Lots of docs added to some of the core components. I'm on my way to the Apple store to have them fix this piece-o-junk, so hopefully I'll have it back soon!


Read more...

Monday, August 17, 2009

Committed all changes

All changes have been committed to P4 (TcpRegression) and Git (PyPcap). Eveyrthing works on my dev box, but we all know that means nothing. I'll sort out any odds and ends tomorrow. I need to get up for work in 3 hours.

- Zach


Read more...

Sunday, August 16, 2009

And so it ends

SoC is coming to an end tonight.

Overall, I think the project went very well. I'm going to try to bust out a few deliverable tests tonight since those were the original goal, but I think the volume of work that I got done will speak for itself.

  • I forked PyPcap, fixed bugs, extended the functionality, wrote better tests.
  • I forked PCS-0.5, fixed bugs, extended the functionality (although I didn't write tests for PCS).
  • I've essentially implemented a reference userland implementation of TCP on top of PCS and PyPcap, which should allow for even more applications than a simple regression testing suite, complete with tests of the code itself, as well as a few deliverable tests that should validate functionality of various TCP stacks.

In addition to continuing work with the TcpRegression suite, I've sent an email to the Metasploit project to see if they could use another leisure-pace developer. That should be an interesting project, and maybe I'll be able to apply some of the knowledge about TCP that I gained over the summer. Overall, it's been very productive - for my personal benefit (fiscal and intellectual), and hopefully the FOSS community and FreeBSD as well.

I've got a few minor changes to PyPcap that need to go up -- mostly it's just the inclusion of a function that will print out a string of bytes in the same format that they appear in Wireshark's Packet Bytes view.

There's a load of TcpRegression functionality and tests that will be in the next commit. Over the next week or so, I'll work on the documentation and cleaning up the code, and getting Google their code sample. I'd like a nice, solid "0.1" release. It's also crossed my mind to separate the regression tests themselves from the main framework, and re-badging it "TCPython". The name looks like it hasn't been taken yet, but we'll see how that goes.

Special thanks to Titus Brown and George Neville-Neil for helping me throughout the summer, you both helped me out of a few ruts along the way that could've made the whole project a lot less enjoyable.


Read more...

Published changes...

Published changes to the PyPcap library. Get the new hotness here:

http://zachriggle.github.com/pypcap/
or
git clone git://github.com/zachriggle/pypcap

This build tested on Mac OS X 10.5.7 and FreeBSD 7.1

CHANGELOG

pypcap-1.2:
- included changes from George Neville-Neil's PCS (http://pcs.sf.net)
- added more tests
- added logging functionality
- implemented patches provided by several users from the original PyPcap site on Google Code. Thanks to...
getxsick
dirk-loss.de

- Changed some interfaces, but made sure to allow backwards compatibility.
Examples:

__init__ now has separate args for interface, filename, and dump file.
dump_close from PCS is now closeDumps
the filter can be set by accessing the '.filter' property
each pcap object can now be re-opened after closing with

- openLive
- openOffline
- openDump

- exposed findalldevs() [getxsick]
- exposed lookupnet() [getxsick]
- exposed 'cnt' arg of loop() [getxsick]
- fixed setnonblock, loop() [dirk-loss.de]


Read more...

Busy Day

Busy day. Added some more PyPcap functionality, fixed some bugs, and implemented tests. All of the functionality is at least touched by the tests...
- Live capture
- Dumping to file
- Reading from file
- Packet filter

Additionally, migrated the 'tcpfilter' class to use bpf instead of manually inspecting the fields of each packet after it had been constructed by PCS. This should be MUCH faster.

Also learned that in Python...

class X():
....y = someObject()

... means that all X objects will be instantiated with the *same instance* of someObject, that is instantiated when the class is declared, instead of each object being allocated when the class instance is created. Weird. I thought it was run when the class was instantiated, and just allowed for the constructor to be a little bit less cluttered. Guess I was wrong (or that there's a Python bug).


Read more...

Thursday, August 13, 2009

C'mon Nose, stop being a stupid WHAAAAAAAAAT

In python...

>>> t = TestPcap('')
>>> t.testEnumerateInterfaces()
>>> t.testOpenDefaultInterface()
>>> t.testOpenSpecificInterface()

But...

zach@Zachs-Computer:~/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests$nosetests testPcap.py
E.E
======================================================================
ERROR: testEnumerateInterfaces (testPcap.TestPcap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 33, in testEnumerateInterfaces
listOfIfs = pcap.findalldevs()
AttributeError: 'module' object has no attribute 'findalldevs'

======================================================================
ERROR: testOpenSpecificInterface (testPcap.TestPcap)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 38, in testOpenSpecificInterface
iface = pcap.findalldevs()[0]
AttributeError: 'module' object has no attribute 'findalldevs'

----------------------------------------------------------------------
Ran 3 tests in 0.004s

FAILED (errors=2)

Here's the output of a "print dir(pcap)" from within the test, run through nosetests.

-------------------- >> begin captured stdout << ---------------------
['DLT_ARCNET', 'DLT_AX25', 'DLT_CHAOS', 'DLT_EN10MB', 'DLT_EN3MB', 'DLT_FDDI', 'DLT_IEEE802', 'DLT_LINUX_SLL', 'DLT_LOOP', 'DLT_NULL', 'DLT_PFLOG', 'DLT_PFSYNC', 'DLT_PPP', 'DLT_PRONET', 'DLT_RAW', 'DLT_SLIP', '__author__', '__builtins__', '__copyright__', '__doc__', '__file__', '__license__', '__maintainer__', '__name__', '__revison__', '__url__', '__version__', 'bpf', 'calendar', 'dltoff', 'ex_name', 'lookupdev', 'pcap', 'sys', 'time']
--------------------- >> end captured stdout << ----------------------

Obviously, it's not there. Pcap is imported as:

try:
import pcs.pcap as pcap
except:
import pcs

Here's the output of "print pcs.__file__" from Python:
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/pcap.so

Aaaand from Nose-tests:
/Library/Python/2.5/site-packages/pcs/pcap.so

Whaaaaaaaaaaat?


Read more...

Time

Lost track of time tonight, stayed up way later than I wanted to. Overhauled a LOT of code in the PyPcap library (both C any Python code). It's now a lot cleaner, and a lot of the functionality that was only available via creating a new pcap object is now available at runtime (i.e. open/reopen an interface or offline capture).

Also changed a bit of the Windows code that (for whatever reason) attempted to implement pcap_lookupdev, even though WinPcap has that function. Other minor changes were included in the C code (a little bit of refactoring).

The Python code now includes logging for most of its functionality, which is disabled by default. It does require either [1] modifying the PYX file or [2] modifying pcap.DEBUG_LEVEL before instantiating the pcap object that logging is desired for. Lots of refactoring. Hopefully I didn't break anything.

Must. Get. To. Sleeeeeeeeeeep.


Read more...

Wednesday, August 12, 2009

Nope, it was Python.

http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html

The __next__ method

Extension types wishing to implement the iterator interface should define a method called __next__, not next. The Python system will automatically supply a next method which calls your__next__. Do NOT explicitly give your type a next method, or bad things could happen.


Read more...

Wtf?

Changing the name of "__next__"... what the hell, Python? (Although I'm much more apt to blame Pyrex)

File "<stdin>", line 1, in <module>

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read
return self.pcapHandle.readpkt()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt
packet = self.read()

File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read
packet = self.file.next()


Read more...

Yep.

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read
return self.pcapHandle.readpkt()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt
packet = self.read()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read
packet = self.file.next()
File "pcap.pyx", line 502, in pcap.pcap.__next__
KeyboardInterrupt


Read more...

Not understanding the logic flow here

Going back and testing stuff. Check this out...

self.filter.read() # tcpregression.tcpfilter.TcpFilter.read()

which calls:

return self.pcapHandle.readpkt() # pcs.PcapConnector.readpkt()

which calls:

packet = self.file.next()[1] # pcs.pcap.pcap.next()

However, debug statements show

"IN __NEXT__".

Dubbleyou Tee Eff.


Read more...

HA!

Finally figured out the logging bit... a few changes to the config file, combined with:

logFiles  = [ 'logging.conf',
join( expanduser( '~' ), 'logging.conf' ),
join( dirname( __file__ ), 'logging.conf' ),
None ]

for logFile in logFiles:
if exists( logFile ):
print "Using logfile %s" % logFile
logging.config.fileConfig( logFile )

Fixed the problem


Read more...

Monday, August 10, 2009

Modem dead

Posting from iPhone. My router & modem fried in storm. :-/ fml Read more...

The Issue [Nevermind, Fixed]

Nevermind, in the process of writing this post, I made a change at some point that made everything work all honky-dory.

build.py in pcs-0.5 and pypcap both use the following command to link everything:

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap_ex.o -L/usr/lib -lpcap -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so

build.py in tcpregression links it with...

/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap_ex.o -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so

The discrepancy between the two being:

-L/usr/lib -lpcap

Exactly why this is the case, I'm unsure. Obviously, if pcap.so isn't linked with libpcap (the actual pcap library), there would be issues.

The build system uses setup.py from pcs-0.5, with a modification to include the different source directory. Here's the diff between pcs' setup.py and my "pcssetup.py". All of the changes have to do with the source being in a "src" folder.

< pcap_cache = 'src/pcs/pcap/config.pkl'
---
> pcap_cache = 'pcs/pcap/config.pkl'
71c71
< f = open( 'src/pcs/pcap/config.h', 'w' )
---
> f = open( 'pcs/pcap/config.h', 'w' )
122c122
< sources = [ 'src/pcs/pcap/pcap.pyx', 'src/pcs/pcap/pcap_ex.c' ],
---
> sources = [ 'pcs/pcap/pcap.pyx', 'pcs/pcap/pcap_ex.c' ],
129a130,140
>
> setup( name = 'pcs',
> version = '0.5',
> description = 'Packet Construction Set',
> author = 'George V. Neville-Neil',
> author_email = 'gnn@neville-neil.com',
> url = 'http://pcs.sf.net',
> packages = ['pcs', 'pcs.packets'],
> cmdclass = pcap_cmds,
> ext_modules = [ pcap ],
> )

And then the actual setup file....

from distutils.core import setup, Extension
import pcssetup
setup( name = 'tcpregression',
version = '1.0',
description = 'FreeBSD TCP Regression Suite',
author = 'Zach Riggle',
author_email = 'zjriggl@freebsd.org',
url = 'http://freebsd.org',
packages = ['tcpregression',
'tcpregression.pcsextension',
'tcpregression.tests',
'pcs',
'pcs.packets'],
package_dir = {'':'src'},
cmdclass = pcssetup.pcap_cmds,
ext_modules = [pcssetup.pcap],
)

As you can see, not too many changes.


Read more...

Proper Building with Setup.py

Thought I'd give proper building with "setup.py" a go. Back to square one with:

ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/build/lib.macosx-10.5-i386-2.6/pcs/pcap.so
Expected in: dynamic lookup

Humbug.


Read more...

Massive Cleanup

I'm going back through a lot of the code, removing commented-out code, deleting un-used files, and restructuring a bit. Hopefully this'll make everything a lot cleaner for when I'm done :-)


Read more...

Sunday, August 9, 2009

Problem Solved

Evidently "setup.py build_ext -i" doesn't do somethign that the regular build operation does. Not sure what. Anyway, the Python documentation for setup.py says that the output files will always end up in build/lib/ or build/lib.arch/ so a "cp build/lib*/pcap.so ..." will take care of it. Hopefully that'll solve the problem.


Read more...

PyPcap issues when running in-directory

Some of the issues I originally had when getting PyPcap setup back in June are re-surfacing, now that I've got PCS and PyPcap (as pcs.pcap) inside of the tcpregression folder:

>>> import pcs.pcap
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pcs/__init__.py", line 68, in <module>
import pcs.pcap as pcap
ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter
Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/src/tcpregression/pcs/pcap.so
Expected in: dynamic lookup


Read more...

Saturday, August 8, 2009

Build system

I've integrated the PyPcap changes (http://zachriggle.github.com/pypcap/) into the TcpRegression Perforce tree, and put the custom-modified PCS-0.5 directory in there, as well. It contains some bugfixes for PCS-0.5, as well as a few modifications to make things easier for my particular uses.

The build system for building PyPcap, and putting the module into the PCS directory is also in place. That was kind of an interesting foray into make and setup.py, but it works very cleanly now that I've got it figured out (thanks Titus!).

Next steps are to write a few quick tests for the TcpRegression library itself, mainly to make sure that the threading stuff works as expected. Assuming that it does, I'll be able to crank out some TCP tests and have a final product for FreeBSD.

My work schedule changes quite a bit next week, so I'll be awake while most people are asleep/at work (depending on how I do it). Basically I'm working 0200-1000 next week, and need to figure out if I'm gonna wake just before 2AM, or go to bed just after 10AM.


Read more...

Thursday, August 6, 2009

GitHub site, setup.py issues

"git push".

Now the stuff is ACTUALLY on the site. My god is the default color scheme ugly. Working to fix that!

http://zachriggle.github.com/pypcap

If anybody is familiar with setup.py, please contact me via email.


Read more...

Wednesday, August 5, 2009

PyPcap Migrated to Git

http://github.com/zachriggle/pypcap

A bunch of changes have been made. I'm working on going back in and documenting those changes. I'll have a changelog shortly, as well as a demo.


Read more...

Forked, Fixed, Updating

Taking some advice and forking PyPcap. Working on adding the patches provided by users. Right now I'm trying to get Pyrex to like the changes that I'm making to make the 'findalldevs' function closer to the real one. A user submitted a patch on the Google Code site, but it doesn't return ALL of the data the function should. Unfortunately, I have to add a bunch of types and unroll several linked-lists and arrays to get all of the data.


Read more...

Sunday, August 2, 2009

Perforce Woes

I don't know if I mentioned this earlier, but Perforce is *by far* the hardest version management system that I've ever had to work with -- out of CVS and SVN, that is.

First off, it defines its operation off of environment variables. This makes manipulation pretty hard unless I define environment variables in bashrc or something -- and changing environments pretty hard to boot.
Second off, it complains every time my hostname changes. Right now, it keeps complaining that I can only commit from "Zachs-Computer", which is the hostname.
Third, I have to define a workspace. This in itself is a pain. I can see the benefits, although if I have to change my hostname to use a workspace, it's kind of moot point. "svn co path/to/server" FTW.

I'm back off to fight with P4 while I try to commit my code :-\


Read more...

PyPcap = Abandonware

So PyPcap, the Pcap library that PCS includes in its distribution, is essentially abandonware. Hasn't been a new release since Jan 2007. The files in the included PyPcap say that GNN is the maintainer.

George, if you are reading this, please let me know so that I may submit patches to PyPcap. There is at least one issue that is of high importance to my project, for which a patch already exists, that I don't mind integrating into the PyPcap project on Google Code.

I tweeted at @dugsong, so hopefully the guy in charge of the project (as far as Google Code is concerned) will give me access to make the changes. I can make the changes locally, but there's no reason not to include them online for all to use.


Read more...

Saturday, August 1, 2009

Okay, so that's done... but we're stuck with a hanging thread [HALP!]

The full TCP exchange of 3-way-handshake, data transfer (with acknowledgement) and active-close (Established -> Fin Wait 1 -> Fin Wait 2 -> Time Wait) has been successfully tested, and works fine.

Here's the script, and subsequent output:

import testconfig
from tcpstatemachine import *
from threading import Thread
from testserver import runServer
from multiprocessing import Process

def test():
t = TcpStateMachine()
t.die = False

serverProcess = Process(target=runServer)
serverProcess.start()

t.open()
t.send('COOOOOOOL!')
t.close()

serverProcess.terminate()

>>> import test
2009-08-01 17:35:18,212 - tcpFilter - INFO - Opened lo0
>>> test.test()
2009-08-01 17:35:19,867 - TcpStateMachine - DEBUG - __init__ - (('127.0.0.3', 31765), ('127.0.0.1', 32261))
2009-08-01 17:35:19,870 - tcpFilter - DEBUG - openInterface - Tried to re-open same interface: lo0
2009-08-01 17:35:19,870 - tcpFilter - INFO - Opened lo0
2009-08-01 17:35:19,870 - TcpStateMachine - INFO - Local host: 127.0.0.3:31765
2009-08-01 17:35:19,871 - TcpStateMachine - INFO - Remote host: 127.0.0.1:32261
~~~~~ In runServer
~~~~~ Listening on 127.0.0.1:32261
~~~~~ Waiting for accepted connection
2009-08-01 17:35:19,950 - TcpStateMachine - DEBUG - open - In valid state to open connection
2009-08-01 17:35:19,967 - TcpStateMachine - DEBUG - open - Generated ISS: 1036344287
2009-08-01 17:35:19,969 - TcpStateMachine - DEBUG - open - Sending SYN packet
2009-08-01 17:35:19,975 - TcpStateMachine - STATE - Setting state from Closed to Syn-Sent
2009-08-01 17:35:19,976 - TcpRecvDaemon - INFO - Starting receive thread for object TcpStateMachine( (127.0.0.3,31765), (127.0.0.1,32261) )
2009-08-01 17:35:19,977 - TcpSendDaemon - INFO - Starting send thread for object TcpStateMachine( (127.0.0.3,31765), (127.0.0.1,32261) )
2009-08-01 17:35:19,980 - TcpStateMachine - DEBUG - open - Waiting for state == ESTABLISHED... Outbound Sequenes: segmentBuffer(limit=4294967296,base=1036344287,copyList=['[SYN]']+[])
2009-08-01 17:35:20,047 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344287 to 1036345633 in ['[SYN]']
2009-08-01 17:35:20,049 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344287 to 1036344288: ['[SYN]']
2009-08-01 17:35:20,050 - TcpSendDaemon - DEBUG - sendThread - Haz a SYN!
2009-08-01 17:35:20,054 - TcpStateMachine - SENT - [<class tcp Seq: 1036344287, 31765->32261, [S], reserved: 0, checksum: 7711, offset: 5, window: 65535L>,
<class payload {payload: ''}>]
2009-08-01 17:35:20,484 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:20,485 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201111, 32261->31765, [AS], reserved: 0, checksum: 65058, offset: 6, window: 65535, ack_number: 1036344288>]
2009-08-01 17:35:20,487 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] IN SYN_SENT
2009-08-01 17:35:20,489 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Has ACK
2009-08-01 17:35:20,490 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Accepted ACK
2009-08-01 17:35:20,492 - TcpStateMachine - DEBUG - debugWithState - [Syn-Sent] Has SYN and either good ACK or no ACK
2009-08-01 17:35:20,492 - TcpStateMachine - STATE - Setting state from Syn-Sent to Established
Sending ack: {seq: 1036344288, ack: 1, ack#: 1883201112}
2009-08-01 17:35:20,497 - TcpStateMachine - SENT - [<class tcp Seq: 1036344288, 31765->32261, [A], reserved: 0, checksum: 20344, offset: 5, window: 65535L, ack_number: 1883201112L>]
~~~~~ Connected
~~~~~ Waiting on Data
2009-08-01 17:35:20,499 - TcpStateMachine - DEBUG - open - Established
2009-08-01 17:35:20,507 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344288 to 1036345634 in ['C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!', 'C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!']
2009-08-01 17:35:20,508 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344288 to 1036344308: ['C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!', 'C', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'L', '!']
2009-08-01 17:35:20,511 - TcpStateMachine - SENT - [<class tcp Seq: 1036344288, 31765->32261, [A], reserved: 0, checksum: 21671, offset: 5, window: 65535L, ack_number: 1883201112L>,
<class payload {payload: 'COOOOOOOL!COOOOOOOL!'}>]
Received Data: COOOOOOOL!COOOOOOOL!
~~~~~ Waiting on Data
2009-08-01 17:35:21,008 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:21,009 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [A], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344308>]
2009-08-01 17:35:21,009 - TcpStateMachine - ERROR - This point should never be reached!
2009-08-01 17:35:21,059 - TcpStateMachine - STATE - Setting state from Established to Fin-Wait-1
2009-08-01 17:35:22,021 - TcpSendDaemon - DEBUG - sendThread - Sending items 1036344308 to 1036345654 in ['[FIN]']
2009-08-01 17:35:22,023 - TcpSendDaemon - DEBUG - sendThread - Sending sequences 1036344308 to 1036344309: ['[FIN]']
2009-08-01 17:35:22,024 - TcpSendDaemon - DEBUG - sendThread - Haz an FIN!
2009-08-01 17:35:22,028 - TcpStateMachine - SENT - [<class tcp Seq: 1036344308, 31765->32261, [AF], reserved: 0, checksum: 20323, offset: 5, window: 65535L, ack_number: 1883201112L>,
<class payload {payload: ''}>]
Received Data:
~~~~~ No more data
2009-08-01 17:35:23,016 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:23,017 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [A], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344309>]
2009-08-01 17:35:23,019 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-1] Our FIN is ACKed
2009-08-01 17:35:23,019 - TcpStateMachine - STATE - Setting state from Fin-Wait-1 to Fin-Wait-2
2009-08-01 17:35:23,020 - TcpStateMachine - ERROR - This point should never be reached!
2009-08-01 17:35:23,023 - TcpStateMachine - INFO - Segment arrived!
2009-08-01 17:35:23,024 - TcpStateMachine - RECVD - [<class tcp Seq: 1883201112, 32261->31765, [AF], reserved: 0, checksum: 65054, offset: 5, window: 65535, ack_number: 1036344309>]
2009-08-01 17:35:23,025 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-2] connection closing
2009-08-01 17:35:23,027 - TcpStateMachine - DEBUG - debugWithState - [Fin-Wait-2] sending ACK of FIN
Sending ack: {seq: 1036344309, ack: 1, ack#: 1883201113}
2009-08-01 17:35:23,029 - TcpStateMachine - SENT - [<class tcp Seq: 1036344309, 31765->32261, [A], reserved: 0, checksum: 20322, offset: 5, window: 65535L, ack_number: 1883201113L>]
2009-08-01 17:35:23,030 - TcpStateMachine - STATE - Setting state from Fin-Wait-2 to Time-Wait
2009-08-01 17:35:23,031 - TcpStateMachine - DEBUG - debugWithState - [Time-Wait] Starting TIME_WAIT timer
2009-08-01 17:35:24,035 - TcpSendDaemon - DEBUG - sendThread - Quitting send thread
2009-08-01 17:35:33,032 - TcpStateMachine - STATE - Setting state from Time-Wait to Closed

It's a bit more verbose than it needs to be, and there's a few things that will be taken out (i.e. "Haz a SYN!") but overall I'm pleased with the way it works.

However, the only remaining problem before the real-real test-writing gets going is that the call to retrieve the next packet from PCAP is a blocking call that's performed by a Python thread (which has no terminate() or equivalent method).

Looking into the issue a bit, pypcap has a "setnonblock(nonblock=True)" method, but due to the way everything is done, this isn't quite so easy to work with (there are also some bug reports that say that this functionality doesn't necessarily work as expected).


Read more...

Friday, July 24, 2009

BitString Cleanup

Finally settled on the below design for BitString. I had added, and then removed, functionality for AND, OR, and XOR because it is ambiguous as to what length to use if the operands are not the same length.

For example, it should be possible to AND
0xFFFF & 0x7
To get 0x7.

However, is that '0b0000000000000111' or '0b0111' or some other variant. Given that the values are indexes from the most-significant-bit, this creates an issue.

Anyway, here it is below. Modding PCS to use this for decoding simple values.

class  BitString(list):
'''
Creates a list containing the individual bits from a byte-string, integer,
or list of values (where each list is interpreted as 1 or 0 based on its
truth balue ).

Bits are ordered from most-significant to least-significant.

>>> import pcs
>>> x = pcs.BitString(0xDEADBEEF)
>>> x
[1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1]
>>> x[4:12]
[1, 1, 1, 0, 1, 0, 1, 0]
>>> int (x)
3735928559L
>>> hex(x)
'0xdeadbeefL'
>>> hex(x[16:32])
'0xbeef'
>>> bin(int (x))
'0b11011110101011011011111011101111'
>>> str (x)
'\xde \xad \xbe \xef '
>>> x[0:8] = 0xAA
>>> x[8:16] = [1,1,1,1,0,0,0,0]
>>> hex(x)
'0xaaf0beefL'
>>> int (x) == 0xAAF0BEEF
True
'''

def __init__(self,bytes):
# Convert integer values to their byte representation
if isinstance(bytes,int) or isinstance(bytes,long):
packString = ''
if abs(bytes) <= 0xff :
packString = '!B'
elif abs(bytes) <= 0xffff :
packString = '!H'
elif abs(bytes) <= 0xffffffff :
packString = '!I'
bytes = struct.pack(packString, bytes)

# Convert a list into a byte representation by taking each item
# and turning it into a bit.
if isinstance(bytes,list) or isinstance(bytes,tuple):
self.length = len(bytes)*8
for index,i in enumerate(bytes):
self.append(1 if i else 0 )

# For strings or string'ized integers, do bit-shifty stuff
# >>> bin(0xFF00)
# '0b1111111100000000'
if isinstance(bytes, str):
self.length = len(bytes)*8
# Do each byte in-order
for i,byte in enumerate(bytes):
# Most-significant-bit first
for bit in range(8 )[::-1 ]:
self.append(1 if ord(byte) & (1 << bit) else 0 )


def __int__(self):
intVal = 0
for index,i in enumerate(self):
intVal += self[-index-1 ] << index
return intVal

def __str__(self):
value = int(self)
formatString = "%0" + ("%s" % (len(self)/8 )) + "x"
hexstring = formatString % value
while len(hexstring) % 2 != 0 :
hexstring = '0' + hexstring
byteString = binascii.unhexlify(hexstring)

while len(byteString) < len(self)/8 :
byteString = "\x00" + byteString
return byteString

def __hex__(self):
return hex(int(self))

def __getitem__(self, x):
return list.__getitem__(self,x)

def __getslice__(self,i,j):
return BitString(list.__getslice__(self,i,j))

def __setitem__(self, i, x):
x = 1 if x else 0
list.__setitem__(self,i,x)

def __setslice__(self,i,j,x):
bs = BitString(x)
list.__setslice__(self,i,j,bs)

def __invert__(self):
return BitString([0 if x else 1 for x in self])

def __add__(self,x):
return BitString(list.__add__(self,x))

Read more...

Thursday, July 23, 2009

Untitled

asdfasdf

class  BitString(object): 
'''
Allows bit-level manipulation of a string as if it were an array.

Example:
>>> x = pcs.BitString('\xff \xff ')
>>> print x
0b1111111111111111
>>> print repr (x.bytes)
'\xff \xff '
>>> x[4:12] = 0
>>> print x
0b1111000000001111
>>> print bin(x[:8])
0b11110000
>>> print bin(x[8:12])
0b0
>>> print bin(x[12:])
0b1111
>>> print repr (x.bytes)
'\xf0\x0f'
'''
def __init__(self, bytes):
self.value = 0
self.length = len(bytes)*8
for i,byte in enumerate(bytes[::-1 ]):
self.value += ord(byte) << (i*8 )
def __getitem__(self, n):
if n > len(self):
raise IndexError
return 1 if self.value & (1 << (len(self)-n-1 )) else 0
def __getslice__(self, i, j):
retVal= 0
i = max(i,0 )
j = min(j,len(self))
for n,bit in enumerate(range(i,j)):
retVal <<= 1
retVal += self[bit]
return retVal
@property
def bytes(self): return self.getBytes()
def getBytes(self):
formatString = "%0" + ("%s" % (len(self)/8 )) + "x"
hexstring = formatString % self.value
byteString = binascii.unhexlify(hexstring)

while len(byteString) < len(self)/8 :
byteString = "\x00" + byteString
return byteString
def __len__(self):
return self.length
def __str__(self):
return self.bin()
def __repr__(self):
return self.__class__.__name__ + "(%s)" % repr(self.getBytes())
def __setitem__(self, n, x):
x = 1 if x else 0
if x:
self.value |= (x << (len(self)-n-1 ))
else :
self.value = self.value & self._makeMask(n)
def __setslice__(self, i, j, x):
i = max(i,0 )
j = min(j,len(self))

isAList = True
if isinstance(x,int) or isinstance(x,long) or isinstance(x,bool):
isAList = False

for index,n in enumerate(range(i,j)):
if isAList:
self[n] = x[index]
else :
self[n] = x
def _makeMask(self, bit):
mask = 0
bitSkip = (len(self)-bit-1 )
for i in range(len(self)):
if i == bitSkip:
continue
mask += 1 << i
return mask
def hex(self):
return hex(self.value)
def bin(self):
return bin(self.value)

When used in conjunction with pcs.Field, I've set it up to calculate the value with the new method, but still *use* the value from the old method, to show the difference between the new vs. old method. Pay particular attention to syn, ack, fin, reset, and push.

>>> synAckPkt = tcp('09\xd4191\xf6\xae\x00\x00\x00\x01`\x12\xff\xff\xfe"\x00\x00')
Decoding sport --> 12345
Decoding dport --> 54321
Decoding sequence --> 959575726
Decoding ack_number --> 1
Decoding offset --> 6
Decoding reserved --> 0
Decoding urgent --> 0
Decoding ack --> 1
Decoding psh --> 0
Decoding reset --> 0
Decoding syn --> 1
Decoding fin --> 0
Decoding window --> 65535
Decoding checksum --> 65058
Decoding urg_pointer --> 0
>>> print repr(synAckPkt)
<class tcp {reset: 4, psh: 2, reserved: 0, sequence: 959575726, ack: 1, checksum: 65058, offset: 6, syn: 9, urgent: 0, window: 65535, ack_number: 1, dport: 54321, sport: 12345, fin: 0, urg_pointer: 0}>
>>> print repr(synAckPkt.data)
<class payload {payload: ''}>

Read more...

Wednesday, July 22, 2009

Just Kooky

So PCS can pick up the fields just fine, as long as they're zeros. Make SYN *and* ACK set to '1' and everything is FUBAR.

Packet data taken from:
http://support.microsoft.com/kb/172983

Syn Packet Data:
TCP: ....S., len: 4, seq: 8221822-8221825, ack: 0, win: 8192, src: 1037
dst: 139 (NBT Session)

Syn-Ack Packet Data:
TCP: .A..S., len: 4, seq: 1109645-1109648, ack: 8221823, win: 8760,
src: 139 (NBT Session) dst: 1037

>>> from pcs.packets.tcp import tcp
>>> import binascii
>>> synAck = "008B040D0010EE8D007D747F60122238012D0000020405B42020"
>>> syn = "040D008B007D747E0000000060022000F2130000020405B42020"
>>>
>>> synPkt = tcp(binascii.a2b_hex(syn))
>>> synAckPkt = tcp(binascii.a2b_hex(synAck))
>>>
>>> print repr(synPkt)
[TCP: reset: 0, reserved: 0, sequence: 8221822, ack: 0, checksum: 61971, offset: 6, syn: 1, urgent: 0, window: 8192, push: 0, ack_number: 0, dport: 139, sport: 1037, fin: 0, urg_pointer: 0]
>>>
>>> print repr(synAckPkt)
[TCP: reset: 4, reserved: 0, sequence: 1109645, ack: 1, checksum: 301, offset: 6, syn: 9, urgent: 0, window: 8760, push: 2, ack_number: 8221823, dport: 1037, sport: 139, fin: 0, urg_pointer: 0] Read more...

Tuesday, July 21, 2009

PCS Bit Encoding

Just some interesting code that I found in PCS. As far as I can tell, the bits of code are the exact same, except for the "shift" field, which is not even used in the scope of the conditional.


# These two bits of code are the same.
elif fieldBR > byteBR:
print "Cannot fit field into current byte"
shift = fieldBR - byteBR
mask = 2 ** byteBR - 1
value = (ord(bytes[curr]) & mask)
fieldBR -= byteBR
byteBR = 8
curr += 1 # next byte
elif fieldBR == byteBR:
mask = 2 ** byteBR - 1
value = ord(bytes[curr]) & mask
fieldBR -= byteBR
byteBR = 8
curr += 1 # next byte
real_value += value << fieldBR
Read more...

Yeah

Working on the aforementioned PCS bug brought to my attention that I've fixed a few bugs in PCS that need to be fixed in the general distribution before running the below script will work.

Rather, try this (which should work in PCS-0.5 without modification):

from pcs.packets.tcp import tcp
t = tcp('09\xd4191\xf6\xae\x00\x00\x00\x01`\x12\xff\xff\xfe"\x00\x00')
print repr(t)
Read more...

PyDev

Just found out that PyDev has a built-in Python shell that does completion.

FML.
Read more...

Full-Stop

Ran into a BIG issue with PCS last night. Evidently it's mis-identifying packet fields for TCP.

Demo included below:

from pcs.packets.localhost import *
import binascii
x = '020000004500002cca844000400600007f0000017f0000033039d4313931f6ae000000016012fffffe220000'
bytes = binascii.a2b_hex(x)
l = localhost(bytes)
ip = l.data
tcp = ip.data
print repr(tcp)
print tcp

Prints out the following. Notice that instead of SYN/ACK, it is identified as SYN/ACK/RST/PSH.
[class tcp {reset: 4, psh: 2, reserved: 0, sequence: 959575726, ack: 1, checksum: 65058, offset: 6, syn: 9, urgent: 0, window: 65535, ack_number: 1, dport: 54321, sport: 12345, fin: 0, urg_pointer: 0},
class payload {payload: ''}]

That packet is a loopback-ipv4-tcp packet, should look like the following:
Transmission Control Protocol, Src Port: italk (12345), Dst Port: 54321 (54321), Seq: 959575726, Ack: 1, Len: 0

Here's the bytes printed in an easier-to-read format. Emphasis on the flags mine.
02 00 00 00 45 00 00 2c ca 84 40 00 40 06 00 00
7f 00 00 01 7f 00 00 03 30 39 d4 31 39 31 f6 ae
00 00 00 01 60 12 ff ff fe 22 00 00

Read more...

Friday, July 17, 2009

Interesting Observation

In Python, None is so truly equivalent to *nothing* that help(None) spawns the interactive help.

I thought that was interesting.


Read more...

Got Money, Got Paid

Finally!

Anyway, I haven't mad a blog post in waaaaaaaaaaaaaay too long. Anyway, the bit that I'm working on now is to resolve the issue of sequence number wrap-around. The way that I'm implementing it will (hopefully) have the benefit of automating the calculation of SND.NXT and a buffer of octets that need to be retransmitted based off of the send/sent segments, and SND.UNA.

Essentially, it's a continuous list with an offset. Less simple than it would originally appear due to the way slices work. Here's a quick example:

Example:
>>> from segmentBuffer import *
>>> sb = segmentBuffer(base=10, max=15)
>>> sb += range(8)
>>> sb[10] # The first item is at [10]
0
>>> sb[11] # The second item is at [11]
1
>>> sb # However, the items are in the proper order
[0, 1, 2, 3, 4, 5, 6, 7]
>>> sb[10:] # And are properly sliced
[0, 1, 2, 3, 4, 5, 6, 7]
>>> sb[10:2] # And the slices wrap-around at max
[0, 1, 2, 3, 4, 5, 6]
>>> sb[2:10] # And slices stop when it hits a gap.
[7]
>>> sb == sb[10:] # Truth works just fine.
True
>>> sb += 2 # The offset is incremented easily
>>> sb # Notice that the first two items are deleted
[2, 3, 4, 5, 6, 7]
>>> len(sb) # The length indeed goes down 8->2
6
>>> sb += range(100,200)
>>> len(sb) # Adding 100 items keeps the correct length
15
>>> sb # Only items up to the limit are added
[2, 3, 4, 5, 6, 7, 100, 101, 102, 103, 104, 105, 106, 107, 108]
>>> sb2 = sb + range(300,400)
>>> sb is sb2 # Addition operator returns a new object.
False
>>> sb2 += 3 # The other object can be incremented
>>> [i for i in sb if i not in sb2]
[2, 3, 4] # Just an example to show the difference
>>> sb[12:15] # This shows that the items were taken off the front
[2, 3, 4]
>>> sb[12:2]
[2, 3, 4, 5, 6]
>>> sb2[12:2] # The indexes for the remaining items do not change
[5, 6] # Although the deleted items are gone.
>>> sb
[2, 3, 4, 5, 6, 7, 100, 101, 102, 103, 104, 105, 106, 107, 108]
>>> sb2
[5, 6, 7, 100, 101, 102, 103, 104, 105, 106, 107, 108]


Read more...

Thursday, July 9, 2009

Birthday and Eclipse Galileo

The birthday went pretty well (I'm still alive, after all) and I just grabbed Galileo and installed all of my tools (PyDev, CDT, Perforce) that I need on a regular basis. Excited to see that there's a Cocoa implementation of Eclipse now, and Galileo seems to start up a lot faster than Ganymeade (don't know if that's an effect of Cocoa-Carbon or general bloat reduction).


Read more...

Monday, July 6, 2009

WTF

Evidently Ecto ate my last SEVERAL blog posts, by not actually posting anything.


Read more...

Hmm


Read more...

No, no, no


Read more...

Sunday, July 5, 2009

Okay, okay, okay


Read more...

I Think I've Got It!

I might have finally figured out the packet flow for the most flexibility and utility.


  1. Packet arrives on interface
  2. Is queued by libpcap
  3. Thread grabs packet off of pcap queue
  4. IF the packet is NOT the packet we are expecting next (seg.sequence != rcv.nxt)
    1. Throw it into a list used by the thread.
    2. Go back up one level
  5. ELSE the packet is one we were expecting
    1. Add the packet to the State Machine's recv() queue, IF there is not already a packet with the same sequence number
    2. IF processing is enabled
      1. Process the packet (i.e. "Segment Arrives" stuff)
      2. Advance the 'processed packet' index, which points to the last packet in the recv() queue that has been processed.
    3. Iterate over all packets in the thread's personal queue. This is done because we might have already received the 'next' packet, but it was received out-of-order.

Okay, so now the packets are being added to a queue, and they are in the correct order. They are also conditionally processed/handled/whatever you want to call it.

Now, we are writing a test. We want to stop when [1] A certain packet is received or [2] A specific state change occurs. This means that two methods need to be hooked into: the state-change method, and the packet-handling method. With the state change, we will want to pass control to the user (i.e. disable auto-processing) AFTER the state transition. With the packet-recv stuff, we want to do it BEFORE the packet is processed. This means that there needs to be a step between '5.1' and '5.2' that checks to see if the packet is flagged, and appropriately disables processing when the packet is encountered.

The test-writer then can call the 'recv()' method and be guaranteed to get the next packet, in the proper order. Additionally, the test-writer can set "break points" that disable auto-processing. This is important, because auto-processing can lead to sending out additional packets (i.e. response to a SYN-ACK, response to a FIN packet, etc.) that may need to be omitted for a test to be successful.

Whew.

Read more...

Saturday, July 4, 2009

Multiprocessing

Hmm. Multiprocessing may not be the way to go, as it runs into issues.
Might have to think about this some more.

Read more...

Implementing TCP Stack Stuff

So I'm implementing more and more of the TCP stack. The idea is that if I actually implement a TCP stack (which was expected, in small chunks) then I [1] understand how TCP should operate much better which allows me to [2] write tests much better and [3] identify corner-cases much better.

One of the issues that I ran into tonight reaches back to my idea to run a separate process that's always "listening" for new packets, and does all of the "Segment Arrives" processing (see pp65 in the RFC).

In order to manipulate the TCP stack of some remote host, it is expected that it will be necessary to get one side of the connection into a specific state (e.g., get the remote state to be SYN-SENT, or the local state to be SYN-RECEIVED) and then perform specific operations that test very specific points of the specification.

In order to do this efficiently, it becomes of the utmost importance to be able to get into these states very easily. Connection buildup and tear-down must be simple, as they are not important except for the individual tests that test these portions (and even then, we might want to skip right to receiving the SYN-ACK packet, before sending the ACK packet to enter the Established state).

In order to perform these state transitions gracefully and properly, it becomes necessary to implement the entire TCP stack, and build in hooks that halt at certain points (on a state transition, specific packet sent or received, malformed packet, etc.).

Now, it turns out that I still hadn't done enough planning to foresee the issue with packet ordering. The model that I thought would work would be to call a "recv()" method that would receive the next packet, and then the test would do something interesting with it. That becomes a problem when the test is expecting a specific field -- say, with a specific ACK number -- but receives that packet after receiving some other packet. The test might not properly realize that the first packet (which didn't match the expected pattern) was pertinent, and would be needed later in the test.

This gives two options:

  1. The recv() method should only return the next packet (i.e. the segment's sequence number == rcv.nxt)
  2. The recv() method simply fetches the next packet off of a queue, to which packets are added in-order by a separate thread/process that manages the state.

Both of these approaches have their limitations. Consider a test in which we want to send a malformed ACK packet in response to a SYN-ACK packet. The code for that might look something like this:

tsm = TcpStateMachine()
i = tsm.packetsRecvd.size()
syn = tsm.newPacket({'syn':1})
tsm.sendRawTcp(syn)
startTime = time.time()
# Keep looping. Note that this loop would be rolled into a seperate method.
while True:
# 30 second timeout. Test fails.
if time.time() < startTime + 30:
return False
# If no new packets we received, sleep
if tsm.packetsRecvd.size() <= i:
time.sleep(0.1)

# Loop through available packets.
for i in range(i, tsm.packetsRecvd.size()):
p = tsm.packetsRecvd[i]
# See if it's a SYN-ACK and the ack number matches.
if p.ack and p.syn and p.ack_number == syn.sequence + 1:
break
# Do stuff with packet at offset 'i'.

The code would send a SYN packet, with the other fields auto-filled, then wait until some packet is received, then do processing based on that.

Actually, after writing that, performing some method of callback on a state transition could lead to un-necessary complications. What might be more useful is a way to turn off auto-processing (not receipt of, just processing of) packets upon some trigger (i.e. after transitioning to state 'FIN-WAIT-1', don't do anything with the packets), and a way to see if that trigger has been hit (although checking to see if the state is FIN-WAIT-1 should be sufficient).

Hmm.

Working all of this out in writing might have cleared things up a bit.

Thanks!


Read more...

Thursday, July 2, 2009

Time Off Did The Trick

Looks like a little time-off did the trick. Got back into the problem that I was working on last (fighting with the operating system, and the various interfaces). Looks like using 127.0.0.X where X is not '1' works to elicit the correct SYN/ACK response, on the correct interface (lo0), and the host OS doesn't try to fight me about it.

Now all I've gotta do is get the test working :-)

Hopefully, things will be smooth sailing from here on forward!


Read more...

Soy Libre!

Just finished up the last of my exams. About friggin' time. I feel pretty confident about all of them -- and I've been doing better this semester than a usually do.

Schedule for the rest of the evening is GSoC, same for tomorrow. Saturday I've got a BBQ for the 4th. GSoC starts accepting status reports on the 6th, they're due on the 13th (which means I have to submit it to Google/GNN, and they/he has to approve it by then).

Should be an interesting weekend.

I turn 21 next Wednesday (the 8th) so count on nothing getting done on the 8th/9th (Maybe on the 9th, I took the day off from work).


Read more...

Sunday, June 28, 2009

Also

Enabling "stealth mode" (which should kill all responses to an unsolicited [from the OS's point of view] SYN/ACK packets) does not prevent the OS from responding on lo0. Evidently stealth mode is not active on the loopback interface.

Grrrrrrr.


Read more...

Power is back on, But network problems continue

Power's back on, woooooooooo.

Okay, so I have two problems:
[1] If I attempt to 'send' packets 'from' an IP address that my computer actually has, the host OS's TCP stack sends a RST packet when it sees a SYN/ACK response to my 'sent' SYN packet.
[2] If I attempt to 'send' packets 'from' a different IP address, the packets get bounced back and forth several times between my computer and the router. For whatever reason, the listening server ('nc -l <port>') doesn't respond at all.

Yay.


Read more...
Power is out. Posting this from my iPhone. Read more...

Fighting with the host OS

After finding out that Wireshark is stupid, and considers 0x2000 to be the same as 0x0002 for Loopback-layer family fields, I am now battling with the host OS. Sending a connection request properly elicits a SYN/ACK response, but the OS is quickly sending out a RST packet to kill the connection.

Changing the source IP to be something other than an IP the operating system should care about results in no SYN/ACK packet. I tried moving back over to a non-loopback interface, with a different source IP address (using the correct destination IP address). It gets sent back from the router (the packet is duplicated with the source MAC being the router MAC), but it does not elicit a SYN/ACK response.

Wierd. And frustrating.


Read more...

Ah.

So, evidently even if you have a host that is listening on 0.0.0.0:12345 (e.g. "nc -l 12345") and inject packets onto an active interface, pointed at that interface's IP address (e.g. "en1" has IP 172.16.0.10) -- any normally-generated packets to/from a local IP address go over "lo0".

That would have been good to know, what, a few hours ago?


Read more...

Saturday, June 27, 2009

Troubleshooting

Lots of troubleshooting the code right now, trying to get the 3-way connect to work. Lots of issues with typos, unfortunately. A lot of it had to do with using (for example) "off" in place of "offset" to access the offset of a TCP packet, because of the way it's declared:

off = pcs.Field( "offset", 4 )

My eyes just saw the first part, and ignored the second half when I started writing code. Thought I went back and fixed all of the fields, evidently not!


Read more...

Friday, June 26, 2009

P4 Is Getting Kind of Annoying

I don't know if it's P4 itself, or P4 integration with Eclipse, but a versioning system should never prevent editing local files because a connection attempt failed, or because the host-name of the machine changed.

That is all.


Read more...

Saturday, June 20, 2009

Checksumming No Se Funciona

Alright, so the last part of the puzzle that needs to fall in place is the checksum. Conveniently, the algorithm for checksumming IP and TCP packets is the same (albeit you feed them different sets of information).

The algorithm works somewhat like so (source IBM):

ushort checksum16(uchar* data, int len)
{
uint sum = 0;
if ((len & 1) == 0)
len = len >> 1;
else
len = (len >> 1) + 1;
while (len > 0) {
sum += *((ushort*)data);
data += sizeof(ushort);
len--;
}
sum = (sum >> 16) + (sum & 0xffff);
sum += (sum >> 16);
return htons(~sum);
}

That is, it sums all of the 16-bit words, does some 1's compliment fanciness, and then returns the 16 least significant bits (inverted). Of course, the operation can be done with any arbitrary multiple of 16 bits. Since most machines nowadays use 32- or 64-bit arithmetic, there's no reason not to do it with 32 bits (or 64). However, the final result is always 16 bits wide.

In Python, the code looks like (or so I think) the following:

def checksum( bytes ):
tmpbytes = bytes
total = 0
# Must be a number of bytes equal to a multiple of two.
if len( tmpbytes ) % 2 == 1:
tmpbytes += "\0"
# For each pair, add the value to the total.
for i in range( len( tmpbytes ) / 2 ):
total += ( unpack( "!H", tmpbytes[( 2 * i ):( 2 * i ) + 2] )[0] )
# Keep shifting
while total >> 16:
total = ( total >> 16 ) + ( total & 0xffff )
# One's complement
total = ~total
# Make into 2 bytes
total = total & 0xffff
# Return value
return total

However, given an IP layer: 0x4500004096da40004006eca1c0a80168d04524e6, the checksum is 0xECA1. When calculating the checksum, the checksum field is set to zero (0x0000). So, the checksum is calculated like so:

>>> from binascii import *
>>> from struct import *
>>> data = unhexlify('4500004096da40004006<b>0000</b>c0a80168d04524e6')
>>> data
'E\x00\x00@\x96\xda@\x00@\x06\x00\x00\xc0\xa8\x01h\xd0E$\xe6'
>>> cksum = checksum(data)
>>> cksum
60577
>>> hex(cksum)
'0xeca1'

Obviously, the end result is incorrect. Turns out that the values that I was using for testing were incorrect. I've since fixed them in the blog post and the offending code.


Read more...

Friday, June 19, 2009

Glad I got up early

So I got up early before class today to work on the problem. I don't know what the issue was, I think it was a matter of incorrectly setting the TCP offset that made Wireshark freak out.

Another PCS Architecture Rant
This time, I'm a bit curious about the design of the pcs.Chain class. It inherits from the 'list' base type, but instead of using itself to store the data, it actually *contains* a seperate list. Example:

>>> isinstance(myChain,list)
True
>>> myChain
[]
>>> myChain.packets
[<ethernet: src:="" \x00\x1bc\x06\x82\xb2',="" dst:="" \x00!)\xa5\xa9?',="" type:="" 2048="">, <ipv4: hlen:="" 5,="" protocol:="" 6,="" src:="" 2886729738l,="" tos:="" 0,="" dst:="" 3494107623l,="" ttl:="" 64,="" length:="" 44,="" version:="" 4,="" flags:="" offset:="" checksum:="" id:="" 56649="">, <tcp: reset:="" 0,="" reserved:="" sequence:="" 3915041816l,="" ack:="" checksum:="" 60177,="" offset:="" 5,="" syn:="" 1,="" urgent:="" window:="" 65535,="" push:="" ack_number:="" 0l,="" dport:="" 80,="" sport:="" 52722,="" fin:="" urg_pointer:="" 0="">, <payload:>]
</payload:>

Not sure what's going on with that. Its constructor only takes a list object:

class  Chain(list):
def __init__(self, packets = None ):
list.__init__(self)
self.packets = packets
self.encode()
# Versus
class Chain(list):
def __init__(self, packets = None ):
list.__init__(self, packets)

This would end up saving a lot of time and make the resulting code look cleaner:

That would make it easier to perform nifty operations. pcs.Chain offers a "checksum" operation, that operates on the whole chain. However, consider the utility of "myChain[-2:].calc_checksum()". Now we can checksum an arbitrary range of packets! Not immediately useful, unless you realize that this is a nifty way to calculate the IP checksum ;-).

Also, I have to wonder about the duplication of functionality offered by the pcs.Chain class, and the pcs.Packet.data member. It seems that, for the large part, they perform the exact same function. Consider:

>>> p = payload()
>>> t = tcp()
>>> i = ipv4()
>>> e = ethernet()
>>> e.data = i
>>> i.data = t
>>> t.data = p
# Finally, these two should have the same ultimate result
>>> e.chain()
>>> chain([e,i,t,p])

I can see the reasoning behind the "data" member -- it makes it easy to set-and-forget what the next level is, and the "chain()" method is a nice helper that builds a chain. Consider the earlier example of myChain[-2:].calc_checksum() currently looks like: myChain.packets[-2].chain().calc_checksum().


Read more...

Thursday, June 18, 2009

Malformed Packet Bull@$!#

Getting pretty sick and tired of Wireshark being useless in telling me exactly why my TCP packets are malformed.
Just sayin'.


Read more...

Saturday, June 13, 2009

Unrelated Note

On an unrelated note, FreeBSD handles pthread_mutex's shared status properly. Mac OS X does not.

Debugging code that relies on this on Mac OSX may cause hair loss.

(This was from my first CSE410 project, thought I'd post it here while I was posting things).


Read more...

Bailed on previous idea

Also of note, I bailed on the idea from last night (and before) of using raw IP sockets to do the sending. The idea was to avoid needing to ever touch the physical or IP layer, thus removing some amount of work (e.g., no setting of the IP length fields, IP checksum calculation).

However, using raw sockets has the caveat of not providing a receive buffer. If using raw IP sockets, one cannot override the 'protocol' field to be IPPROTO_TCP. Haven't put much more thought into using raw TCP sockets.

Short Rant on pcs.Field
The class pcs.Field is used as a base class to represent any field inside of a packet. It provides bit-width flexibility, and the ability to encode/decode values of arbitrary bit-width. This is very useful when defining the various fields of a packet, as you can specify their exact size, and then with the pcs.Layout class, specify their exact order.

However, in order to string these two things together, you get somewhat of a kluge. Consider the following code in tcp.py:

     def  __init__(self, bytes = None ):
"""initialize a TCP packet"""
sport = pcs.Field("sport" , 16 )
dport = pcs.Field("dport" , 16 )

And then the corresponding code to manipulate those fields...

>>> from pcs.packets.tcp import tcp
>>> t = tcp()
>>> type(t.offset)
<type 'int'>
>>> type(t.sport)
<type 'int'>

Okay, great. In the __init__ method, they were working on local variables with the same name as the variables to be used. The pcs.Field objects are supplied with a name to correspond to the object property (e.g. the pcs.Field object with (... name="sport",...) corresponds to the actual field "pcs.Field.sport". Great.

However, the approach falls short here. pcs.Field provides no way of setting the bit-order of whatever field it is manipulating, so users are stuck doing this:

t.sport = socket.htons(80)
# Instead of
t.sport = 80

And having the "magic" just happen. This prevents simple comparisons, as well. Which of the following is easier to read?

if ntohs(t.sport) == 80: ...
if t.sport == htons(80): ...
if t.sport == 20480: ...
if t.sport == 80: ...

This is less trivially simple for fields that may have more than one representation. This was discussed in a previous post (IIRC) with relation to IP and Hardware addresses. Consider the following. The IP address "127.0.0.1" can be represented as:

  • 16777343 (host-byte-order)
  • 2130706433 (network-byte-order)
  • '\x01\x00\x00\x7f' (host-byte-order byte-string)
  • '\x7f\x00\x00\x01' (network-byte-order byte-string)
  • And of course, "127.0.0.1"

Obviously, this makes things a bit difficult to work with. However, the current method does not just use network-byte order. An IP address is a NBO long. A port is a NBO short. An ethernet address is a NBO byte-string. Figuring out what the proper internal representation for everything is a pain, and translating between HBO and NBO makes it worse.

The simple solution is to allow multiple ways of getting and setting a field. Example using ports (also works with IP addresses, ethernet addresses):

>>> t.sport.setAscii('80')
>>> t.sport.setInteger(80)
>>> t.sport.setNetworkInteger(20480)
>>> t.sport.setBytes('P\x00\x00\x00')
>>> t.sport.setNetworkBytes('\x00\x00\x00P')
>>> t.sport == 80
True
>>> t.sport.getInteger() == 80
True
>>> t.sport == '80'
True
>>> t.sport == 'P\x00\x00\x00'
True
>>> t.sport == '\x00\x00\x00P'
False
>>> t.sport.getNetworkBytes() == '\x00\x00\x00P'
True

This allows us all the flexibility in the world that is needed. Assuming that all of the different types support the base functions (get/set for ASCII, Integer, and Bytes in both host- and network-byte order), everything is quickly extendible.

Visiting the original code, things have improved slightly. No longer do we need to worry about bit-ordering.

# Old
t.sport = htons(80)
t.sport = htons(str('80'))
t.sport = struct.unpack('!L','\x00\x00\x00P')
# New
t.sport.setInteger(80)
t.sport.setAscii('80')
t.sport.setNetworkBytes('\x00\x00\x00P')

Alright, now we face the problem of making the encoding 'cleaner'. Encoding things is extremely simple if everything is aligned on an 8-bit byte boundary. One could simply use "t.sport.getNetworkBytes() + t.dport.getNetworkBytes()" ad infinitum. However, that approach is ugly. Overriding __add__() with some gentle application of isinstance() and you can get the bytes pretty easily (even without 8-bit byte boundaries).

Will do some work tonight on this front. Will make for a very worthwhile patch to PCS.

[/rant]


Read more...

If Red Bull would just...

...sponsor Google Summer of Code, it would make my life much easier (and less expensive).
Read more...

Works pretty well

After moving everything to Python 2.6 (read: re-installing Pyrex, PyLint, PyLibPcap, and PCS), the packet-queueing process was extremely easy to do. Much easier than I thought it would be. Here's an example of it in action (note that the TCP data that is interpreted by PCS is whatever the equivalent of the ASCII data is).

Note that the output gets a little bit wacky as both processes fight for StdOut.

Python 2.6.2 (r262:71600, May  2 2009, 17:25:25)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from packetQueue import *
>>> ipl = IpListener("127.0.0.1")
>>> ipl.process.start()
>>>
>>> from socket import *
>>> host = gethostbyname(gethostname())
>>> port = 0
>>> text = "Hello"
>>>
>>> def send(data=text):
... sender = socket(AF_INET,SOCK_RAW,IPPROTO_IP)
... sender.sendto(data, (host,port))
...
>>> send()
>>> send(Hello
'This is a test')
>>>
>>> ipl.recvQueue.get()
This is a test
<TCP: reset: 0, reserved: 0, dport: 27756, sequence: 1862270976, ack: 0, checksum: 0, ack_number: 0, syn: 0, urgent: 0, window: 0, offset: 0, push: 0, sport: 18533, fin: 0, urg_pointer: 0>
>>> ipl.recvQueue.get()
<TCP: reset: 29, reserved: 13, dport: 26995, sequence: 543781664, ack: 7, checksum: 0, ack_number: 1629516901, syn: 58, urgent: 3, window: 0, offset: 7, push: 14, sport: 21608, fin: 0, urg_pointer: 0>
>>>
>>> ipl.process.terminate()


Read more...

First Round Done

Not really a project update, but a worthwhile note: the first round of exams are done at MSU, and I have already completed my second-semester project for my CSE course. There's a paper due on the 25th, and the homework for all of the next week is already done. Should be able to put in some extra time this weekend and the upcoming week. Right now is about 3 weeks from the end of the semester, then my it's hyperspeed.


Read more...

Thursday, June 11, 2009

Although...

Although there IS the catch that if we do not call recv() before the response TCP packet arrives, it will be lost forever (since we are using raw sockets). The easy solution to this is to have a thread that does all of the buffering, as I imagine happens in a real TCP stack. Might have to look into the 'multiprocessing' module. It seems like passing pcs.packets.tcp.tcp objects via a multiprocessing.queue would be the way to go about doing things.

Actually, after reading a bit about the multiprocessing module, this may end up being a non-issue. Just a have a reader thread that reads all of the raw IP data, checks to see if it is the right stuff (i.e. the IP addresses are what were expected, the TCP ports are the expected ports) and then passes it along to the queue. Behind-the-scenes, a call to recv() will just fetch the next item off the queue.


Read more...

Working with Raw IP

I figure that messing with IP stuff at any point in time is a Bad Idea (TM), so doing anything at all with the pcs IP or Ethernet layer shouldn't be done. Ideally, I should be able to point an IP packet (filled with TCP data and payload) at some IP address, and everything will get sorted out.

I figure it'll work something like the code below (need root, open two Terminals, call recv() before send()). Not sure that much more robustness is needed...

from socket import *
host = gethostbyname(gethostname())
port = 0
text = "Hello"
def send():
sender = socket(AF_INET,SOCK_RAW,IPPROTO_IP)
sender.sendto(text, (host,port))
def recv():
recver = socket(AF_INET,SOCK_RAW,IPPROTO_IP)
data = recver.recv(1024)
print data[20:]


Read more...

Sunday, June 7, 2009

Sexy

Is what pylint is.
Integration with PyDev (Eclipse) makes it even better.


Read more...

Thursday, June 4, 2009

Hi, George

George

Looks like there's been an issue again...

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

YOU@neville-neil.com

Message will be retried for 2 more day(s)
...
For help, please quote incident ID 13500304. (state 18).


Read more...

Sunday, May 31, 2009

Logic Validation

Looking for some input on folks familiar with low-level TCP. I've got some general questions about proper values of Sequence # and Ack # over the course of a TCP session.

ACKing Packets

A segment on the retransmission queue is fully acknowledged if the sum of its sequence number and length is less or equal than the acknowledgment value in the incoming segment.

A packet is received with sequence number "seqn" and TCP payload with "len" octets of data. The receiving party should compare "seqn" to the variable outlined in the RFC, "rcv.nxt". If they match, "rcv.next" is incremented by "len".

The ACK number of outgoing packets is always sent to 'rcv.next'-1.

What is done with packets received out-of-order, but that are entirely valid? Buffer them, then check each one's SEQ number when rcv.nxt is incremented?

Delayed ACK response
This also raises the question of how to properly ACK a received message, when packets aren't already being set. In some instances, it may be necessary to send an empty packet with the ACK flag set. How to determine when this needs to be done?


Read more...

IP and Port Representation

Small hurdle that I tried to tackle tonight is the representation of IP addresses and ports. There are at least 4 representations of each that will need to be dealt with regularly:

- ASCII (e.g. "127.0.0.1", "80")
- Host-byte-order integer, useful for ports (16777343, 80)
- Network-byte-order integer, used in raw packets (2130706433, 20480)
- Network-byte-order byte-string, used by "inet_aton" and the like. ('\x7f\x00\x00\x01', '\x00\x00\x00P')

So, the IP address and Port were encapsulated in classes. This should make dealing with them slightly easier.


Read more...

Oh.

I was having some issues earlier wrapping my mind around why, exactly, packing strings didn't give me a string with the "\x##\x##\x##" format. Evidently, that is only necessary for characters that do not have a normal, printable equivalent.

Example:

>>> "\x7D" + "\x7B"
'}{'

Anyway, the following output is what had me confused, and why I spent about an hour or so now looking into the 'struct' package and why it wasn't giving me what I thought I wanted. Notice the plaintext "test" at the end. It seemed out of place, so I was assuming that I was doing something wrong.

>>> from pcsextension.payload import payload
>>> from pcs.packets.tcp import tcp
>>> p = payload('test')
>>> t = tcp()
>>> t.data = p
>>> t.chain().bytes
'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00test'


Read more...

Rejected Email

George

If you're reading this, I'm having some issues contacting you via email:

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

gnn@neville-neil.com

Message will be retried for 2 more day(s)

Technical details of temporary failure:
Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 451 451 4.3.0 Message held for human verification before permitting delivery. For help, please quote incident ID 13393043. (state 18).


Read more...

Monday, May 25, 2009

Memorial Day

Happy Memorial Day, All!


Read more...

Sunday, May 24, 2009

Logging Framework Change

Decided I was going about it all wrong (it was getting late). Changed the implementation to simply provide convenience methods on top of the Python 'logging' package's framework -- with the added benefit of not having to worry about the nitty-gritty of the 'logging' package. Just simple, direct access to the log methods.


>>> from loggable import Loggable
>>> class A(object):
... def __init__(self):
... self.log = Loggable (self)
...
>>> a = A()
>>> a.log.info('test')
2009-05-24 12:48:08,654 - __main__.A - INFO - test
>>> a.log.critical('test')
2009-05-24 12:48:19,475 - __main__.A - CRITICAL - test
>>> a.log.state('test')
2009-05-24 12:48:22,978 - __main__.A - STATE_CHANGE - test
>>> a.log.pktsent ('test')
2009-05-24 12:48:26,402 - __main__.A - PACKET_SENT - test
>>> a.log.generated('test')
2009-05-24 12:48:49,516 - __main__.A - RESPONSE_GEN - test
>>> a.log.field('test')
2009-05-24 12:48:52,475 - __main__.A - FIELD_CHANGE - test


Also have been working on a simple TCP State Machine today. My theory is that we will need to know [1] what state the local TCP connection is in, and [2] what state the remote TCP connection should be in. A simple state machine should allow us to do this with some flexibility. Here is the base class/interface. Suggestions are highly welcomed:

class  TcpState(object):
'''
Base TCP State class. Note that it will, by design, ignore all non-TCP layers.
The TCP State classes will not perform verification that the packets are all
to/from the same hosts. That filtering will be implemented elsewhere in the
framework.
'''

def __init__(self):
self._nextState = None

def packetSent(self, tcpPacket):
'''
Hand off a packet to the TcpState object, where the object will perform
any necessary calculation, and determine if a state change should be
caused by the provided packet.

The packet should be a packet that is considered 'sent' from
the TCP stack that this state belongs to.

@return True if the packet causes a state change. False otherwise.
'''
return False

def packetRecvd(self, tcpPacket):
'''
Same as packetSent, but in the opposite direction.

The packet should be a packet that is considered 'received' from
the remote TCP stack.

@see packetSent
'''
return False

def causedStateChange(self):
'''
Returns true if a previous call to packetSent/packetRecvd caused a
state-change.
'''
return self._nextState != None

def getNextState(self):
'''
Returns the class (subclass of TcpState) that represents the state that
the TCP stack should be in, as a result of previous packetSent/Recvd calls.
@return A class object, or None if no state change should occur.
'''
return self._nextState

Interesting Predicament: Blogo does not like pasting HTML code inside of "pre" tags. It tries to make the HTML code visible, by changing each < and > to "<" or ">", respectively. It seems that I can't find a Blog client that does everything perfectly. So no Python syntax highlighting for now.

Read more...

Logging Framework

I think I've got the Logging stuff set up the way I'd like it to work.

Here's an example (the doc for the responsible class)...

    >>> from loggable import Loggable
>>> Loggable().logfield('test')
2009-05-24 03:23:40,851 - loggable.Loggable - FIELD_CHANGE - test
>>> Loggable().loginfo('test')
2009-05-24 03:24:26,108 - loggable.Loggable - INFO - test
>>> class B(Loggable):
... pass
...
>>> B().loginfo('test')
2009-05-24 03:24:38,306 - __main__.B - INFO - test
>>> B().logpacket('packet XYZ sent')
2009-05-24 03:26:05,299 - __main__.B - PACKET_SENT - packet XYZ sent
>>> B().logresponse('response ABC generated for packet XYZ')
2009-05-24 03:26:15,745 - __main__.B - RESPONSE_GEN - response ABC generated for packet XYZ
>>> B().logfield('field abc123 set to newValue')
2009-05-24 03:26:42,292 - __main__.B - FIELD_CHANGE - field abc123 set to newValue
>>> Loggable().logstate('state changed to CONNECTED')
2009-05-24 03:31:16,848 - loggable.Loggable - STATE_CHANGE - state changed to CONNECTED


Read more...

Further thoughts on design

A few more thoughts on the design of the framework:

  1. Logging (textual)
  2. Logging (packets)
  3. Diff'ing (packets)
  4. Live history

Logging (textual)
The way that the textual output of the framework is delivered needs to be well-thought-out. While it's obvious that "test passed" / "test failed" is all that should be necessary for standard runs (INFO level), there are many distinct levels below INFO that will be necessary. A few of these off the top of my head:

  • Full packet sent'd/recv'd over network IF
    This is obvious. Just a Packet sent/recv, with the BinHex print-out

  • State-change
    This should somewhat reflect the diagram provided on Page 23 of RFC 793. This behavior will also make it easier to track issues in the flow.
    1. Closed
    2. Listen
    3. SYN sent
    4. SYN recvd
    5. Established
    6. Fin Wait-1/2
    7. Close Wait
    8. Closing
    9. Time Wait
    10. Closed
  • Response generation
    Automated response generation (i.e. Send packet X, recv packet Y. Validate that Packet Y is what we were expecting, and then generate Packet Z that is a valid response/ACK for Packet Y).

  • Field generation (dynamic)
    When sending information, some fields (like IP length, checksums) may be generated dynamically/on-the-fly.

  • Manual field override
    Testing invalid states/data will also be necessary. Manually overriding fields (e.g. bad checksum) should provide extra insight. This might belong on the same logging level as Field Generation, but with a different prefix.

Logging (Packets)
Creating a TCPDUMP-formatted file with the contents of all of the packets from a test session will be invaluable in determining what, specifically, went wrong during the tests. This will make the use of third-party tools possible, and should complement the configurable logging well.

Differential Analysis
Simple packet replay, will only be useful in select circumstances. Ideally, sending packet X will always generate response Y (with some variables, such as timestamp). Playback of a TCPDUMPed session would allow this, but it would need a framework to determine what fields are important for validity, and which aren't. Since it would be difficult to require the specific lack of a particular response, this application may be very limited.

Live History
Having a history (in memory) of all of the packets sent/received during the current test will be a necessity. This is required to make sure that, for example, all of the packets were properly ack'ed as we continue sending packets. Also, it will be necessary to have internal state information about how much data has been ack'ed, and its correlation to the window size.


Read more...

Followers

Blog Archive