projects

In this area you'll find other things I'm working on and/or interested in.

But have I mentioned my beautiful son lately?

---

File under projects Tue Oct 4 08:09:03 CT 2005

Form philosophy

A form management package has one goal: to help preserve the integrity of data as it moves from server to client and back again.

Most form packages do two things: validate data and serialize data as (X)HTML. Some offer additional client-side validation checks via Javascript, etc. Others offer tight integration with particular data models.

Rose::HTML::Objects does both things well. RHTMLO allows you to define form classes that represent reasonably complicated data models, providing validation and serialization.

Some developers are of the opinion that serialization is not properly the function of a form manager because it blurs the line between view and model. I disagree. Proper and correct serialization is important to the validation process, and hence vital to the model. It is but one step in a series of validation layers.

Validation happens when a human being completes a (X)HTML form, as she self-monitors her attempt to enter good and correct data ("did I spell that correctly?"). Validation may happen again before the form is submitted, using client-side Javascript. Validation happens again when the server receives the request. Again, when the form object is initialized with the submitted data. Again, when the data is committed to storage. At every step checks are made to preserve what the client submitted and verify that the data conforms with what is expected and required.

Since serialization to and from (X)HTML is part of the roundtrip all data takes, a good form management tool should be able to handle (X)HTML creation as well as server-side validation. That's not blurring the model/view line; it's reflecting the reality that data must be handled by human beings and the web browser is one of the best tools we currently have.

File under projects Thu Aug 14 21:38:36 CT 2008

Wide Finder

An intriguing experiment in testing parallel architecture.

File under projects Mon Jul 28 22:16:43 CT 2008

Years pass, code grows, expectations ... ?

It's the OSCON time of year again. A couple of years ago I was in a hotel in Portland OR, coding up a KinoSearch implementation of Swish3. That was then. Two years later, Swish3 is really no further on, except that it has been largely refactored and is a much more stable, yet alpha, project.

Just remembering that tonight as I update the Perl bindings of Swish3 to use the new tokenization routines. It's the kind of project that moves in fits and starts. I would have liked to have finished it years ago, but it would not be as good, since many of the refactorings I have made over the last couple of years have been the direct result of coding strategies I have learned in my $job(s) and other FOSS projects.

And I find I like my life this way, chipping away at building a better mousetrap while I sip New Belgium beer and listen to Robert Plant and Alison Krauss. Sure, it's a sultry July evening, and my entire life occasionally floats through my mind like a grainy day at the beach, but hey: you only get this chance once, and building a better mousetrap is not a bad way to pass some idle moments.

File under projects Thu Jul 24 21:46:09 CT 2008

Program like it's 1975

I benefited from this article.

File under projects Tue May 6 21:35:18 CT 2008

Swish3 Status 19 April 2008

There's been quite a bit of activity in the last month.

• The C++ Xapian example now can search as well as index, and there are Perl equivalents using Search::Xapian checked into svn as well. The C++ code will read/write the swish.xml header; the Perl does not (yet).
• The meta/prop id unique check now uses a hash for quick look up.
• The test suite for libswish3 is totally restructured. Now using Perl's Test::Harness and added a slew of new meta/prop tests. Alongside that were additions to the NamedBuffer debugging output to print each substring in the buffer.
• Several new string-related utility functions for converting ints to strings and back. Also a new config hash for configration options that use a StringList instead of a simple string.
• Fixed some mem leaks in the example .c programs and added more info to the swish_lint usage() output (including reminders about the various SWISH_DEBUG* env var values).

There are still several parser features yet to be implemented to support the Swish-e 2.4 config options, but those will likely take a backseat to getting a working swish3 Perl script running with SWISH::Prog and SWISH::Prog::Xapian.

File under projects Sat Apr 19 22:42:14 CT 2008

Ruby on Rails

I do not use it myself, preferring Perl+Catalyst. But this is an interesting perspective on deployment issues at a major shared hosting provider and as such, is not limited to RoR.

File under projects Fri Apr 11 09:06:46 CT 2008

Swish3 Status 30 March 2008

More progress with Swish3.

• There is now a swish_xapian.cpp C++ example for using libswish3 with a Xapian backend. All that is complete is the indexing portion; still TODO is the search part. Still, a significant thing that it was so easy to build a search engine.
• The swish.xml header format is complete and can now read/write the header file. Need to add that part to the swish_xapian.cpp example.
• Squashed some long-standing memory leaks when using the filehandle functions.

Little by little.

File under projects Sun Mar 30 22:03:48 CT 2008

Swish3 Status 2008-03-15

I've finally gotten back to Swish3 development after several months away. Hard to believe I've been working on this project for something close to 3 years now.

Lately I have been focusing on the following things:

Header file format

Because Swish3 will have multiple IR backends, it is important that there be a consistent index metadata file that describes the MetaNames, Properties, and tokenizing information, just like the Swish2 header does. Just as with the config file format, it makes sense to define the header file format as XML, since we already have a robust XML parser for free. To make it simple, I have defined the header file XML schema to be the same as the config file schema. In short, you configure Swish3 by creating a header file. The "real" header file will be more strict about explicitly naming all the expected attribute values, numbering the MetaName/Property ids, etc. But the idea is simple: a single XML schema.

I have written the code to read header/config file format and create a swish_Config object. There's also code for merging 2 swish_Config objects together, so that you can define a config file to override an existing header file.

Still TODO is the code for writing the header file.

The Perl bindings have been updated to reflect the new swish_Config API. This required a great deal of reworking and thinking about the Perl API. I had to rewrite things a few times to get a workable solution. The key Perl mantra is "objects on demand." I.e., do not define any Perl objects that wrap C pointers and try to keep them on the XS side. Instead, create all Perl objects "just in time" as part of the get_* method call. This makes reference counting much simpler.

MetaNames and PropertyNames

These now have their own C API with swish_MetaName and swish_Property structs. These relate directly to the header file format and swish_Config. There will end up being a separate PropertyName API for search results. I still think we're going to have to port the Swish2 PropertyNames storage/retrieval code to Swish3 and have a backend-indepedent index.prop file. The issue with this is going to be scaling. One other thought I've had is storing properties in a SQLite db. That route won't allow for presorted properties, but does have the advantage of being much more transparent and de-buggable.

SWISH::Prog

I have moved the SWISH::Prog svn tree to svn.swish-e.org from peknet.com. I also moved SWISH::Filter (and likely will eventually move SWISH::API::More and its cousins).

SWISH::Prog will form the framework for the Perl implementation of Swish3. I know there are some folks who don't like the idea of Swish3 being so Perl-centric. To that I can say only, tough luck. :)

Seriously though, my perspective is that there will be multiple Swish3 implementations. The one I am working on is in Perl using SWISH::Prog. There's nothing to stop someone from implementing one in C or C++ or Java or whatever. libswish3 provides the parsing/tokenizing piece missing from other IR projects, and it is a library for the very reason that implementing a Swish3 program should be language-neutral. If you can link against a C library, then you can write a Swish3 program. The header file API is well documented; the backend is supposed to be pluggable. It's all about the API.

I do intend to write a swish_xapian.cpp program eventually, showing how to implement a C++ Swish3 program with Xapian. That could be the fallback program if you really don't want to use Perl.

Documentation

I've stared a swish_intro.7 and swish_migration.7 set of docs. swish_intro will outline the aggregator/normalizer/analyzer/indexer/searcher philosophy and the outline of the libswish3 API. swish_migration will discuss differences in Swish2 vs Swish3 and how you can convert your config files and move to using Swish3.

File under projects Sat Mar 15 22:23:14 CT 2008

HTTP Flow

This flowchart url posted today on #catalyst.

File under projects Tue Mar 11 09:16:26 CT 2008