tag:blogger.com,1999:blog-69517333023312030112024-03-08T14:28:02.375-05:00gsoc-tcpregressionZachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.comBlogger96125tag:blogger.com,1999:blog-6951733302331203011.post-20466312981927185622009-11-16T21:22:00.002-05:002009-11-16T21:24:38.345-05:00Holy Crap How Time FliesLots of time has gone by since I've had a chance to put some hours into the TCP Regression framework. I haven't forgotten it, just been busy. A trip to New York City, followed by Halloween, and two <b>absolutely gorgeous</b> November sundays kept me away from the computer for the past few weeks.<div><br /></div><div>The coming weekends, since I'm not going home for Thanksgiving, as well as Christmas-time (since most of the other interns leave Dec. 17, and I am staying until Dec. 31) will probably see a spark of productivity.</div>Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-32608192115645774642009-10-11T14:33:00.002-04:002009-10-11T14:40:06.793-04:00Build Issues when InstallingIn the process of investigating the build issues on Mac OS X 10.6 Snow Leopard, I decided to build on FreeBSD.<div><br /></div><div>The combo:</div><div>python setup.py config</div><div>python setup.py build</div><div>sudo python setup.py test</div><div><br /></div><div>Works just fine. However, once the package is installed... things go haywire (full output below). The issue appears to be in the final linking stage.</div><div><br /></div><div><b>make</b></div><div><div>cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o<b><i> -L/usr/lib -lpcap</i></b> -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so</div><div><br /></div><div><b>make install</b></div><div><div>cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so</div><div><br /></div></div></div><div>For whatever reason, libpcap isn't being linked in. The tests attempt to import pcap, then make sure that it is valid, and if that doesn't work, import pcs.pcap (as seen at the end). When attempting to import the installed version of pcap, the "import pcap" statement fails. Grrrr...</div><div><br /></div><div><div><b>[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ ls</b></div><div>CHANGES<span class="Apple-tab-span" style="white-space:pre"> </span>LICENSE<span class="Apple-tab-span" style="white-space:pre"> </span>Makefile<span class="Apple-tab-span" style="white-space:pre"> </span>README<span class="Apple-tab-span" style="white-space:pre"> </span>pcap.c<span class="Apple-tab-span" style="white-space:pre"> </span>pcap.pyx<span class="Apple-tab-span" style="white-space:pre"> </span>pcap_ex.c<span class="Apple-tab-span" style="white-space:pre"> </span>pcap_ex.h<span class="Apple-tab-span" style="white-space:pre"> </span>setup.py<span class="Apple-tab-span" style="white-space:pre"> </span>tests<span class="Apple-tab-span" style="white-space:pre"> </span>testsniff.py</div><div><b>[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ make</b></div><div>pyrexc pcap.pyx</div><div>python setup.py config </div><div>running config</div><div>found {'libraries': ['pcap'], 'library_dirs': ['/usr/lib'], 'include_dirs': ['/usr/include']}</div><div>python setup.py build</div><div>running build</div><div>running build_ext</div><div>pyrexc pcap.pyx --> pcap.c</div><div>building 'pcap' extension</div><div>creating build</div><div>creating build/temp.freebsd-7.1-RELEASE-i386-2.5</div><div>cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o</div><div>pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':</div><div>pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type</div><div>pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':</div><div>pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type</div><div>pcap.c: In function '__pyx_f_4pcap_4pcap_dump':</div><div>pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type</div><div>cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/include -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o</div><div>pcap_ex.c: In function 'pcap_ex_next':</div><div>pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type</div><div>creating build/lib.freebsd-7.1-RELEASE-i386-2.5</div><div>cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -L/usr/lib -lpcap -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so</div><div><b>[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test</b></div><div>python setup.py test</div><div>running test</div><div>testDispatch (tests.testPcap.TestPcap) ... ok</div><div>testEnumerateInterfaces (tests.testPcap.TestPcap) ... ok</div><div>testErrors (tests.testPcap.TestPcap) ... ok</div><div>testInjectedPacketIsReceived (tests.testPcap.TestPcap) ... ok</div><div>testIter (tests.testPcap.TestPcap) ... ok</div><div>testIterable (tests.testPcap.TestPcap) ... ok</div><div>testOpenDefaultInterface (tests.testPcap.TestPcap) ... ok</div><div>testOpenLive (tests.testPcap.TestPcap) ... ok</div><div>testPacketFilter (tests.testPcap.TestPcap) ... ok</div><div>testProperties (tests.testPcap.TestPcap) ... ok</div><div>testReadpkts (tests.testPcap.TestPcap) ... ok</div><div>testWriteDump_ReadOffline (tests.testPcap.TestPcap) ... ok</div><div><br /></div><div>----------------------------------------------------------------------</div><div>Ran 12 tests in 1.541s</div><div><b>[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make install</b></div><div>python setup.py install</div><div>running install</div><div>running build</div><div>running build_ext</div><div>pyrexc pcap.pyx --> pcap.c</div><div>building 'pcap' extension</div><div>cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o</div><div>pcap.c: In function '__pyx_f_4pcap_4pcap_dispatch':</div><div>pcap.c:2298: warning: passing argument 3 of 'pcap_dispatch' from incompatible pointer type</div><div>pcap.c: In function '__pyx_f_4pcap_4pcap_nextPacket':</div><div>pcap.c:2381: warning: passing argument 3 of 'pcap_ex_next' from incompatible pointer type</div><div>pcap.c: In function '__pyx_f_4pcap_4pcap_dump':</div><div>pcap.c:2902: warning: passing argument 1 of 'pcap_dump' from incompatible pointer type</div><div>cc -fno-strict-aliasing -DNDEBUG -O2 -fno-strict-aliasing -pipe -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x20000 -fPIC -I/usr/local/include/python2.5 -c pcap_ex.c -o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o</div><div>pcap_ex.c: In function 'pcap_ex_next':</div><div>pcap_ex.c:272: warning: passing argument 3 of 'pcap_next_ex' from incompatible pointer type</div><div>cc -shared -pthread build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap.o build/temp.freebsd-7.1-RELEASE-i386-2.5/pcap_ex.o -o build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so</div><div>running install_lib</div><div>copying build/lib.freebsd-7.1-RELEASE-i386-2.5/pcap.so -> /usr/local/lib/python2.5/site-packages</div><div>running install_egg_info</div><div>Removing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info</div><div>Writing /usr/local/lib/python2.5/site-packages/pcap-1.2-py2.5.egg-info</div><div><b>[zjriggl@freebsd ~/zachriggle-pypcap-1f0484c]$ sudo make test</b></div><div>python setup.py test</div><div>running test</div><div>Traceback (most recent call last):</div><div> File "setup.py", line 120, in <module></module></div><div> provides = ['pcap'] )</div><div> File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup</div><div> dist.run_commands()</div><div> File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands</div><div> self.run_command(cmd)</div><div> File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command</div><div> cmd_obj.run()</div><div> File "setup.py", line 91, in run</div><div> from tests.testPcap import PcapTestSuite</div><div> File "/usr/home/zjriggl/zachriggle-pypcap-1f0484c/tests/testPcap.py", line 17, in <module></module></div><div> import pcs.pcap as pcap</div><div>ImportError: No module named pcs.pcap</div><div>*** Error code 1</div><div><br /></div><div>Stop in /usr/home/zjriggl/zachriggle-pypcap-1f0484c.</div><div><br /></div><div><br /></div><div><br /></div></div>Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-30093332576255111032009-10-11T12:35:00.003-04:002009-10-11T12:55:35.199-04:00PCS-0.5 Diff with PCS version used in TcpRegressionI've created a Diff file that shows all of the changes that I made to PCS over the course of the summer. Some of these might be merged back into the PCS mainline, others might not.<br /><br />The highlight of the changes are:<br /><ul><li>BitString class. The code for encoding individual bits, even if there is a lack of alignment on 8-bit boundaries, is a bit iffy (even the code says "This makes my head hurt"). The BitString class allows manipulation of individual bits just like a Python list object, and can be translated into an integer value, string value, binary string (via bin()), hex string (via hex()), and allows arbitrary manipulation.</li><li>The names of the pcs.packets.tcp.tcp class fields were assigned variables. Instead of object['fieldName'] which is prone to typos going undetected, I made a bunch of variables so that they can be referred to as object[f_fieldname]. If mistyped, the interpreter picks up on the nonexistent variable.</li></ul><div>GNN: I've explicitly emailed you about this, as this was something that you requested.</div><div><br /></div><div>Everyone else can get a copy here:</div><div><a href="http://p4db.freebsd.org/fileDownLoad.cgi?FSPC=//depot/projects/soc2009/zjriggl%5ftcpregression/diff/pcs%2d0.5.diff&REV=1">http://p4db.freebsd.org/fileDownLoad.cgi?FSPC=//depot/projects/soc2009/zjriggl%5ftcpregression/diff/pcs%2d0.5.diff&REV=1</a></div><div><br /></div><div>From looking at the diff, it does not appear that my changes rely on anything in the forked PCap library (and for that matter, don't involve PCap at all).</div><div><br /></div><div>Zach</div>Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-72053187419341555082009-09-20T18:25:00.003-04:002009-09-20T18:26:14.679-04:00Yep, 10.6 Did It (Methinks)The build.py script was only set to look in the first include directory for pcap.h. On 10.6, /usr/include/pcap.h includes /usr/local/pcap/pcap.h, which contains all of the definitions the build system was looking for.<br /><br />Fixed the build script, will add it to Git soon.Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-89139197828324953452009-09-13T19:42:00.000-04:002009-09-13T19:43:13.408-04:0010.6 Breaks PyPcap?I think Snow Leopard breaks PyPcap -- for whatever reason, I'm getting compile errors that didn't previously exist. I'll have to boot up one of the VMs and see if everything still works OK, or if there was some change that I didn't properly test that I also completely forgot that I made in the first place.Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-83026812840499802532009-08-29T11:49:00.001-04:002009-08-29T11:49:42.844-04:00Docs Added<p style="clear: both">Lots of docs added to some of the core components. I'm on my way to the Apple store to have them fix this piece-o-junk, so hopefully I'll have it back soon!</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-74471323010851758742009-08-17T03:04:00.001-04:002009-08-17T03:04:06.012-04:00Committed all changes<p style="clear: both">All changes have been committed to P4 (TcpRegression) and Git (PyPcap). Eveyrthing works on my dev box, but we all know that means nothing. I'll sort out any odds and ends tomorrow. I need to get up for work in 3 hours.</p><p style="clear: both">- Zach</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-46144696745899264392009-08-16T23:28:00.001-04:002009-08-16T23:29:42.431-04:00And so it ends<p style="clear: both">SoC is coming to an end tonight. </p><p style="clear: both">Overall, I think the project went very well. I'm going to try to bust out a few deliverable tests tonight since those were the original goal, but I think the volume of work that I got done will speak for itself. </p><ul style="clear: both"><li>I forked PyPcap, fixed bugs, extended the functionality, wrote better tests. <br /></li><li>I forked PCS-0.5, fixed bugs, extended the functionality (although I didn't write tests for PCS). <br /></li><li>I've essentially implemented a reference userland implementation of TCP on top of PCS and PyPcap, which should allow for even more applications than a simple regression testing suite, complete with tests of the code itself, as well as a few deliverable tests that should validate functionality of various TCP stacks.<br /></li></ul><p style="clear: both">In addition to continuing work with the TcpRegression suite, I've sent an email to the Metasploit project to see if they could use another leisure-pace developer. That should be an interesting project, and maybe I'll be able to apply some of the knowledge about TCP that I gained over the summer. Overall, it's been very productive - for my personal benefit (fiscal and intellectual), and hopefully the FOSS community and FreeBSD as well.</p><p style="clear: both"></p><p style="clear: both">I've got a few minor changes to PyPcap that need to go up -- mostly it's just the inclusion of a function that will print out a string of bytes in the same format that they appear in Wireshark's Packet Bytes view.</p><p style="clear: both">There's a load of TcpRegression functionality and tests that will be in the next commit. Over the next week or so, I'll work on the documentation and cleaning up the code, and getting Google their code sample. I'd like a nice, solid "0.1" release. It's also crossed my mind to separate the regression tests themselves from the main framework, and re-badging it "TCPython". The name looks like it hasn't been taken yet, but we'll see how that goes.</p><p style="clear: both">Special thanks to Titus Brown and George Neville-Neil for helping me throughout the summer, you both helped me out of a few ruts along the way that could've made the whole project a lot less enjoyable.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-17991761954432416762009-08-16T06:12:00.001-04:002009-08-16T07:06:47.965-04:00Published changes...<p style="clear: both">Published changes to the PyPcap library. Get the new hotness here: <br /><a href="http://zachriggle.github.com/pypcap/" target="_blank"></a></p><p style="clear: both"><a href="http://zachriggle.github.com/pypcap/" style="text-decoration: none;" target="_blank">http://zachriggle.github.com/pypcap/</a><br /><u>or</u><br />git clone git://github.com/zachriggle/pypcap</p><p style="clear: both"><em>This build tested on Mac OS X 10.5.7 and FreeBSD 7.1</em></p><p style="clear: both"><u>CHANGELOG</u></p><p style="clear: both">pypcap-1.2: <br />- included changes from George Neville-Neil's PCS (http://pcs.sf.net) <br />- added more tests <br />- added logging functionality <br />- implemented patches provided by several users from the original PyPcap site on Google Code. Thanks to... <br /> getxsick <br /> dirk-loss.de </p><p style="clear: both">- Changed some interfaces, but made sure to allow backwards compatibility. <br /> Examples: </p><p style="clear: both"> __init__ now has separate args for interface, filename, and dump file. <br /> dump_close from PCS is now closeDumps <br /> the filter can be set by accessing the '.filter' property<br /> each pcap object can now be re-opened after closing with </p><p style="clear: both"> - openLive <br /> - openOffline<br /> - openDump </p><p style="clear: both">- exposed findalldevs() [getxsick]<br />- exposed lookupnet() [getxsick] <br />- exposed 'cnt' arg of loop() [getxsick] <br />- fixed setnonblock, loop() [dirk-loss.de] </p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-47350216918482718872009-08-16T05:04:00.001-04:002009-08-16T05:04:19.862-04:00Busy Day<p style="clear: both">Busy day. Added some more PyPcap functionality, fixed some bugs, and implemented tests. All of the functionality is at least touched by the tests...<br />- Live capture<br />- Dumping to file<br />- Reading from file<br />- Packet filter</p><p style="clear: both">Additionally, migrated the 'tcpfilter' class to use bpf instead of manually inspecting the fields of each packet after it had been constructed by PCS. This should be MUCH faster. </p><p style="clear: both">Also learned that in Python...</p><blockquote style="clear: both"><p>class X():<br />....y = someObject()</p></blockquote><p style="clear: both">... means that all X objects will be instantiated with the *same instance* of someObject, that is instantiated when the class is declared, instead of each object being allocated when the class instance is created. Weird. I thought it was run when the class was instantiated, and just allowed for the constructor to be a little bit less cluttered. Guess I was wrong (or that there's a Python bug).</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-67098692195332324952009-08-13T22:17:00.001-04:002009-08-13T22:17:21.117-04:00C'mon Nose, stop being a stupid WHAAAAAAAAAT<p style="clear: both">In python...</p><p style="clear: both">>>> t = TestPcap('')<br />>>> t.testEnumerateInterfaces()<br />>>> t.testOpenDefaultInterface()<br />>>> t.testOpenSpecificInterface()</p><p style="clear: both">But...</p><p style="clear: both">zach@Zachs-Computer:~/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests$nosetests testPcap.py <br />E.E<br />======================================================================<br />ERROR: testEnumerateInterfaces (testPcap.TestPcap)<br />----------------------------------------------------------------------<br />Traceback (most recent call last):<br /> File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 33, in testEnumerateInterfaces<br /> listOfIfs = pcap.findalldevs()<br />AttributeError: 'module' object has no attribute 'findalldevs'</p><p style="clear: both">======================================================================<br />ERROR: testOpenSpecificInterface (testPcap.TestPcap)<br />----------------------------------------------------------------------<br />Traceback (most recent call last):<br /> File "/Users/zach/Documents/workspace/zjriggl_tcpregression/src/pcs/pcap/tests/testPcap.py", line 38, in testOpenSpecificInterface<br /> iface = pcap.findalldevs()[0]<br />AttributeError: 'module' object has no attribute 'findalldevs'</p><p style="clear: both">----------------------------------------------------------------------<br />Ran 3 tests in 0.004s</p><p style="clear: both">FAILED (errors=2)<br /><br /></p><p style="clear: both">Here's the output of a "print dir(pcap)" from within the test, run through nosetests.</p><p style="clear: both">-------------------- >> begin captured stdout << ---------------------<br />['DLT_ARCNET', 'DLT_AX25', 'DLT_CHAOS', 'DLT_EN10MB', 'DLT_EN3MB', 'DLT_FDDI', 'DLT_IEEE802', 'DLT_LINUX_SLL', 'DLT_LOOP', 'DLT_NULL', 'DLT_PFLOG', 'DLT_PFSYNC', 'DLT_PPP', 'DLT_PRONET', 'DLT_RAW', 'DLT_SLIP', '__author__', '__builtins__', '__copyright__', '__doc__', '__file__', '__license__', '__maintainer__', '__name__', '__revison__', '__url__', '__version__', 'bpf', 'calendar', 'dltoff', 'ex_name', 'lookupdev', 'pcap', 'sys', 'time']<br />--------------------- >> end captured stdout << ----------------------</p><p style="clear: both">Obviously, it's not there. Pcap is imported as:</p><p style="clear: both">try:<br /> import pcs.pcap as pcap<br />except:<br /> import pcs</p><p style="clear: both">Here's the output of "print pcs.__file__" from Python:<br />/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/pcap.so</p><p style="clear: both">Aaaand from Nose-tests:<br />/Library/Python/2.5/site-packages/pcs/pcap.so</p><p style="clear: both"><strong>Whaaaaaaaaaaat?</strong></p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com1tag:blogger.com,1999:blog-6951733302331203011.post-47197264110367081792009-08-13T01:25:00.001-04:002009-08-13T01:25:18.247-04:00Time<p style="clear: both">Lost track of time tonight, stayed up way later than I wanted to. Overhauled a LOT of code in the PyPcap library (both C any Python code). It's now a lot cleaner, and a lot of the functionality that was only available via creating a new pcap object is now available at runtime (i.e. open/reopen an interface or offline capture).</p><p style="clear: both">Also changed a bit of the Windows code that (for whatever reason) attempted to implement pcap_lookupdev, even though WinPcap has that function. Other minor changes were included in the C code (a little bit of refactoring).</p><p style="clear: both">The Python code now includes logging for most of its functionality, which is <strong>disabled</strong> by default. It does require either [1] modifying the PYX file or [2] modifying pcap.DEBUG_LEVEL before instantiating the pcap object that logging is desired for. Lots of refactoring. Hopefully I didn't break anything.</p><p style="clear: both">Must. Get. To. Sleeeeeeeeeeep.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-47703924302101594572009-08-12T21:31:00.001-04:002009-08-12T21:31:07.622-04:00Nope, it was Python.<p style="clear: both"><a href="http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html" target="_blank">http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/version/Doc/Manual/special_methods.html</a></p><p style="clear: both"><strong>The __next__ method </strong></p><p style="clear: both">Extension types wishing to implement the iterator interface should define a method called <strong>__next__</strong>, <em>not</em> next. The Python system will automatically supply a next method which calls your__next__. <strong>Do NOT explicitly give your type a next method</strong>, or bad things could happen.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-19140172370385431772009-08-12T21:29:00.001-04:002009-08-12T21:29:29.225-04:00Wtf?<p style="clear: both">Changing the name of "__next__"... what the hell, Python? (Although I'm much more apt to blame Pyrex)</p><p style="clear: both"> File "<stdin>", line 1, in <module></p><p style="clear: both"> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read<br /> return self.pcapHandle.readpkt()</p><p style="clear: both"> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt<br /> packet = self.read()</p><p style="clear: both"> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read<br /> packet = self.file.next()</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-13961952546870464572009-08-12T21:12:00.001-04:002009-08-12T21:12:32.479-04:00Yep.<p style="clear: both">Traceback (most recent call last):<br /> File "<stdin>", line 1, in <module><br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/tcpregression/tcpfilter.py", line 66, in read<br /> return self.pcapHandle.readpkt()<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 1009, in readpkt<br /> packet = self.read()<br /> File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pcs/__init__.py", line 991, in read<br /> packet = self.file.next()<br /> File "pcap.pyx", line 502, in pcap.pcap.__next__<br />KeyboardInterrupt</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-88686080277098484702009-08-12T20:59:00.001-04:002009-08-12T20:59:43.535-04:00Not understanding the logic flow here<p style="clear: both">Going back and testing stuff. Check this out...</p><p style="clear: both">self.filter.read() # tcpregression.tcpfilter.TcpFilter.read()</p><p style="clear: both">which calls:</p><p style="clear: both">return self.pcapHandle.readpkt() # pcs.PcapConnector.readpkt()</p><p style="clear: both">which calls:</p><p style="clear: both">packet = self.file.next()[1] # pcs.pcap.pcap.next()</p><p style="clear: both">However, debug statements show</p><p style="clear: both">"IN __NEXT__".</p><p style="clear: both">Dubbleyou Tee Eff.</p><p style="clear: both"></p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-57692443571403328912009-08-12T18:11:00.003-04:002009-08-12T18:11:52.425-04:00HA!<p style="clear: both">Finally figured out the logging bit... a few changes to the config file, combined with:</p><pre style="clear: both">logFiles = [ 'logging.conf',<br /> join( expanduser( '~' ), 'logging.conf' ), <br /> join( dirname( __file__ ), 'logging.conf' ), <br /> None ] <br /> <br />for logFile in logFiles: <br /> if exists( logFile ): <br /> print "Using logfile %s" % logFile <br /> logging.config.fileConfig( logFile ) <br /></pre><p style="clear: both">Fixed the problem</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-22236997525241436502009-08-10T22:34:00.001-04:002009-08-10T22:36:17.338-04:00Modem deadPosting from iPhone. My router & modem fried in storm. :-/ fmlZachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-78886680703803126152009-08-10T21:11:00.001-04:002009-08-10T21:11:37.088-04:00The Issue [Nevermind, Fixed]<p style="clear: both"><strong>Nevermind, in the process of writing this post, I made a change at some point that made everything work all honky-dory.<br /></strong><br />build.py in pcs-0.5 and pypcap both use the following command to link everything:</p><blockquote style="clear: both"><p>/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/pcs/pcap/pcap_ex.o -L/usr/lib -lpcap -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so</p></blockquote><p style="clear: both">build.py in tcpregression links it with...</p><blockquote style="clear: both"><p>/usr/bin/gcc-4.0 -L/opt/local/lib -bundle -undefined dynamic_lookup build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap.o build/temp.macosx-10.5-i386-2.6/src/pcs/pcap/pcap_ex.o -o build/lib.macosx-10.5-i386-2.6/pcs/pcap.so</p></blockquote><p style="clear: both">The discrepancy between the two being:</p><blockquote style="clear: both"><p><strong> -L/usr/lib -lpcap </strong><br /></p></blockquote><p style="clear: both">Exactly why this is the case, I'm unsure. Obviously, if pcap.so isn't linked with libpcap (the actual pcap library), there would be issues.</p><p style="clear: both">The build system uses setup.py from pcs-0.5, with a modification to include the different source directory. Here's the diff between pcs' setup.py and my "pcssetup.py". All of the changes have to do with the source being in a "src" folder.</p><blockquote style="clear: both"><p><pre style="clear: both">< pcap_cache = 'src/pcs/pcap/config.pkl'<br />---<br />> pcap_cache = 'pcs/pcap/config.pkl'<br />71c71<br />< f = open( 'src/pcs/pcap/config.h', 'w' )<br />---<br />> f = open( 'pcs/pcap/config.h', 'w' )<br />122c122<br />< sources = [ 'src/pcs/pcap/pcap.pyx', 'src/pcs/pcap/pcap_ex.c' ],<br />---<br />> sources = [ 'pcs/pcap/pcap.pyx', 'pcs/pcap/pcap_ex.c' ],<br />129a130,140<br />> <br />> setup( name = 'pcs',<br />> version = '0.5',<br />> description = 'Packet Construction Set',<br />> author = 'George V. Neville-Neil',<br />> author_email = 'gnn@neville-neil.com',<br />> url = 'http://pcs.sf.net',<br />> packages = ['pcs', 'pcs.packets'],<br />> cmdclass = pcap_cmds,<br />> ext_modules = [ pcap ],<br />> )<br /></pre></p></blockquote><p style="clear: both">And then the actual setup file....</p><blockquote style="clear: both"><p>from distutils.core import setup, Extension<br />import pcssetup<br />setup( name = 'tcpregression',<br /> version = '1.0',<br /> description = 'FreeBSD TCP Regression Suite',<br /> author = 'Zach Riggle',<br /> author_email = 'zjriggl@freebsd.org',<br /> url = 'http://freebsd.org',<br /> packages = ['tcpregression',<br /> 'tcpregression.pcsextension',<br /> 'tcpregression.tests',<br /> 'pcs',<br /> 'pcs.packets'],<br /> package_dir = {'':'src'},<br /> cmdclass = pcssetup.pcap_cmds,<br /> ext_modules = [pcssetup.pcap],<br /> )<br /></p></blockquote><p style="clear: both">As you can see, not too many changes.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-55419671798795938822009-08-10T18:07:00.001-04:002009-08-10T18:07:36.242-04:00Proper Building with Setup.py<p style="clear: both">Thought I'd give proper building with "setup.py" a go. Back to square one with:</p><p style="clear: both">ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter<br /> Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/build/lib.macosx-10.5-i386-2.6/pcs/pcap.so<br /> Expected in: dynamic lookup</p><p style="clear: both">Humbug.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-9539007208065932912009-08-10T00:03:00.001-04:002009-08-10T00:03:22.345-04:00Massive Cleanup<p style="clear: both">I'm going back through a lot of the code, removing commented-out code, deleting un-used files, and restructuring a bit. Hopefully this'll make everything a lot cleaner for when I'm done :-)</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-25633930215793741402009-08-09T22:32:00.001-04:002009-08-09T22:32:42.275-04:00Problem Solved<p style="clear: both">Evidently "setup.py build_ext -i" doesn't do somethign that the regular build operation does. Not sure what. Anyway, the Python documentation for setup.py says that the output files will always end up in build/lib/ or build/lib.<em>arch</em>/ so a "cp build/lib*/pcap.so ..." will take care of it. Hopefully that'll solve the problem.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-18155855359448057282009-08-09T21:32:00.001-04:002009-08-09T21:32:36.263-04:00PyPcap issues when running in-directory<p style="clear: both">Some of the issues I originally had when getting PyPcap setup back in June are re-surfacing, now that I've got PCS and PyPcap (as pcs.pcap) inside of the tcpregression folder:</p><p style="clear: both">>>> import pcs.pcap<br />Traceback (most recent call last):<br /> File "<stdin>", line 1, in <module><br /> File "pcs/__init__.py", line 68, in <module><br /> import pcs.pcap as pcap<br />ImportError: dlopen(pcs/pcap.so, 2): Symbol not found: _bpf_filter<br /> Referenced from: /Users/zach/Documents/workspace/zjriggl_tcpregression/src/tcpregression/pcs/pcap.so<br /> Expected in: dynamic lookup</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-10108859149807159832009-08-08T20:37:00.001-04:002009-08-08T20:37:45.006-04:00Build system<p style="clear: both">I've integrated the PyPcap changes (<a href="http://zachriggle.github.com/pypcap/" target="_blank">http://zachriggle.github.com/pypcap/</a>) into the TcpRegression Perforce tree, and put the custom-modified PCS-0.5 directory in there, as well. It contains some bugfixes for PCS-0.5, as well as a few modifications to make things easier for my particular uses.</p><p style="clear: both">The build system for building PyPcap, and putting the module into the PCS directory is also in place. That was kind of an interesting foray into make and setup.py, but it works very cleanly now that I've got it figured out (thanks <a href="http://ivory.idyll.org/blog" target="_blank">Titus</a>!).</p><p style="clear: both">Next steps are to write a few quick tests for the TcpRegression library itself, mainly to make sure that the threading stuff works as expected. Assuming that it does, I'll be able to crank out some TCP tests and have a final product for FreeBSD.</p><p style="clear: both">My work schedule changes quite a bit next week, so I'll be awake while most people are asleep/at work (depending on how I do it). Basically I'm working 0200-1000 next week, and need to figure out if I'm gonna wake just before 2AM, or go to bed just after 10AM.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0tag:blogger.com,1999:blog-6951733302331203011.post-69396925658294650982009-08-06T21:16:00.001-04:002009-08-06T21:16:34.064-04:00GitHub site, setup.py issues<p style="clear: both">"git push".</p><p style="clear: both">Now the stuff is ACTUALLY on the site. My god is the default color scheme ugly. <s>Working to fix that!</s></p><p style="clear: both"><a href="" title="http://zachriggle.github.com/pypcap">http://zachriggle.github.com/pypcap</a></p><p style="clear: both">If anybody is familiar with setup.py, please contact me via email.</p><br class='final-break' style='clear: both' />Zachhttp://www.blogger.com/profile/03008329063902532712noreply@blogger.com0