Build Issues when Installing

In the process of investigating the build issues on Mac OS X 10.6 Snow Leopard, I decided to build on FreeBSD.

The combo:

python setup.py config

python setup.py build

sudo python setup.py test

Works just fine. However, once the package is installed... things go haywire (full output below). The issue appears to be in the final linking stage.

make

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

make install

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

For whatever reason, libpcap isn't being linked in. The tests attempt to import pcap, then make sure that it is valid, and if that doesn't work, import pcs.pcap (as seen at the end). When attempting to import the installed version of pcap, the "import pcap" statement fails. Grrrr...

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ ls

CHANGES LICENSE Makefile README pcap.c pcap.pyx pcap_ex.c pcap_ex.h setup.py tests testsniff.py

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ make

pyrexc pcap.pyx

python setup.py config

running config

found {'libraries': ['pcap'], 'library_dirs': ['/usr/lib'], 'include_dirs': ['/usr/include']}

python setup.py build

running build

running build_ext

pyrexc pcap.pyx --> pcap.c

building 'pcap' extension

creating build

creating build/temp.freebsd-7.1-RELEASE-i386-2.5

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o

pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':

pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':

pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_dump':

pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o

pcap_ex.c: In function 'pcap_ex_next':

pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type

creating build/lib.freebsd-7.1-RELEASE-i386-2.5

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test

python setup.py test

running test

testDispatch (tests.testPcap.TestPcap) ... ok

testEnumerateInterfaces (tests.testPcap.TestPcap) ... ok

testErrors (tests.testPcap.TestPcap) ... ok

testInjectedPacketIsReceived (tests.testPcap.TestPcap) ... ok

testIter (tests.testPcap.TestPcap) ... ok

testIterable (tests.testPcap.TestPcap) ... ok

testOpenDefaultInterface (tests.testPcap.TestPcap) ... ok

testOpenLive (tests.testPcap.TestPcap) ... ok

testPacketFilter (tests.testPcap.TestPcap) ... ok

testProperties (tests.testPcap.TestPcap) ... ok

testReadpkts (tests.testPcap.TestPcap) ... ok

testWriteDump_ReadOffline (tests.testPcap.TestPcap) ... ok

----------------------------------------------------------------------

Ran 12 tests in 1.541s

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make install

python setup.py install

running install

running build

running build_ext

pyrexc pcap.pyx --> pcap.c

building 'pcap' extension

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o

pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':

pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':

pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type

pcap.c: In function '__pyx_f_4pcap_4pcap_dump':

pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type

cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o

pcap_ex.c: In function 'pcap_ex_next':

pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type

cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so

running install_lib

copying build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so -> /usr/local/lib/python2.5/site-packages

running install_egg_info

Removing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info

Writing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info

[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test

python setup.py test

running test

Traceback (most recent call last):

File "setup.py", line 120, in

provides = ['pcap'] )

File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup

dist.run_commands()

File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands

self.run_command(cmd)

File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command

cmd_obj.run()

File "setup.py", line 91, in run

from tests.testPcap import PcapTestSuite

File "/usr/home/zjriggl/zachriggle-pypcap-1f0484c/tests/testPcap.py", line 17, in

import pcs.pcap as pcap

ImportError: No module named pcs.pcap

*** Error code 1

Stop in /usr/home/zjriggl/zachriggle-pypcap-1f0484c.

Read more...

PCS-0.5 Diff with PCS version used in TcpRegression

I've created a Diff file that shows all of the changes that I made to PCS over the course of the summer. Some of these might be merged back into the PCS mainline, others might not.

The highlight of the changes are:

• BitString class. The code for encoding individual bits, even if there is a lack of alignment on 8-bit boundaries, is a bit iffy (even the code says "This makes my head hurt"). The BitString class allows manipulation of individual bits just like a Python list object, and can be translated into an integer value, string value, binary string (via bin()), hex string (via hex()), and allows arbitrary manipulation.
• The names of the pcs.packets.tcp.tcp class fields were assigned variables. Instead of object['fieldName'] which is prone to typos going undetected, I made a bunch of variables so that they can be referred to as object[f_fieldname]. If mistyped, the interpreter picks up on the nonexistent variable.

GNN: I've explicitly emailed you about this, as this was something that you requested.

Everyone else can get a copy here:

http://p4db.freebsd.org/fileDownLoad.cgi?FSPC=//depot/projects/soc2009/zjriggl%5ftcpregression/diff/pcs%2d0.5.diff&REV=1

From looking at the diff, it does not appear that my changes rely on anything in the forked PCap library (and for that matter, don't involve PCap at all).

Zach

Read more...

Yep, 10.6 Did It (Methinks)

The build.py script was only set to look in the first include directory for pcap.h. On 10.6, /usr/include/pcap.h includes /usr/local/pcap/pcap.h, which contains all of the definitions the build system was looking for.

Fixed the build script, will add it to Git soon. Read more...

10.6 Breaks PyPcap?

I think Snow Leopard breaks PyPcap -- for whatever reason, I'm getting compile errors that didn't previously exist. I'll have to boot up one of the VMs and see if everything still works OK, or if there was some change that I didn't properly test that I also completely forgot that I made in the first place. Read more...

Docs Added

Lots of docs added to some of the core components. I'm on my way to the Apple store to have them fix this piece-o-junk, so hopefully I'll have it back soon!

Read more...

Committed all changes

All changes have been committed to P4 (TcpRegression) and Git (PyPcap). Eveyrthing works on my dev box, but we all know that means nothing. I'll sort out any odds and ends tomorrow. I need to get up for work in 3 hours.

- Zach

Read more...

And so it ends

SoC is coming to an end tonight.

Overall, I think the project went very well. I'm going to try to bust out a few deliverable tests tonight since those were the original goal, but I think the volume of work that I got done will speak for itself.

• I forked PyPcap, fixed bugs, extended the functionality, wrote better tests.
  
• I forked PCS-0.5, fixed bugs, extended the functionality (although I didn't write tests for PCS).
  
• I've essentially implemented a reference userland implementation of TCP on top of PCS and PyPcap, which should allow for even more applications than a simple regression testing suite, complete with tests of the code itself, as well as a few deliverable tests that should validate functionality of various TCP stacks.
  

In addition to continuing work with the TcpRegression suite, I've sent an email to the Metasploit project to see if they could use another leisure-pace developer. That should be an interesting project, and maybe I'll be able to apply some of the knowledge about TCP that I gained over the summer. Overall, it's been very productive - for my personal benefit (fiscal and intellectual), and hopefully the FOSS community and FreeBSD as well.

I've got a few minor changes to PyPcap that need to go up -- mostly it's just the inclusion of a function that will print out a string of bytes in the same format that they appear in Wireshark's Packet Bytes view.

There's a load of TcpRegression functionality and tests that will be in the next commit. Over the next week or so, I'll work on the documentation and cleaning up the code, and getting Google their code sample. I'd like a nice, solid "0.1" release. It's also crossed my mind to separate the regression tests themselves from the main framework, and re-badging it "TCPython". The name looks like it hasn't been taken yet, but we'll see how that goes.

Special thanks to Titus Brown and George Neville-Neil for helping me throughout the summer, you both helped me out of a few ruts along the way that could've made the whole project a lot less enjoyable.

Read more...