Gallery has a new website! It's been many months in the making and
has taken a great deal of time from many people. We chose Drupal as our content management
system and have transitioned all the data from our old website over to
a shiny, new extremely fast server. The road to this new site has
been long and rocky at times. Read more to hear the story of what we
did and how we did it.
In the beginning...
The Gallery website has been around in one incarnation or another
since about 2000. In the early days, it was just couple of HTML pages
on an old server in a closet behind a DSL line. As the project
started to grow and more people got involved, it was a drag to keep
updating the HTML by hand so in December of 2001 we cast around and
found PostNuke. A full-on content
management system was far superior to simple HTML pages in that it
enabled us to have many site administrators with varying degress of
permissions. This lightened the load on our main developers and freed
them up to spend more time writing Gallery code.
As the project grew, the website became more and more popular.
Eventually we had to move it off the DSL line and onto a colocated
server better suited to handle the site's resource requirements. As
we made improvements we found that every time we reduced the latency
on the website we received significantly more traffic, so optimizing
the site became a major goal for us. By August of 2005 the site
handled roughly 2.7M page views from 515K unique visitors. This is
not an exceptionally large amount of traffic, but for a site where the
majority of the traffic is interactive forums it meant that we had to
continually tune the site to keep the latency down. In 2001 Postnuke
was the best CMS available but as we started having to tweak aspects
of it to do get what we needed we essentially created a fork of
Postnuke that was very challenging to maintain.
One of the main problems with Postnuke was that much of our data was
stored in the embedded best-of-breed phpBB forum software. When we
originally started using Postnuke we were using phpBB1.4, which came
with the app by default. But we rapidly hit limits in the early
version (and had to regularly apply security patches) so we took the
hit, installed phpBB2 using the PNphpBB2 module and ported all the
forum data forward using one-off custom migration scripts. But even
though phpBB2 was a significant improvement over phpBB1.4, we still
had the problem that the forums data was essentially in a separate
application that only looked like it was part of the same site.
There was no way to have a forum post appear on the front page of the
site, which meant that any comments on news stories added would wind
up in a separate system from forum posts. Search integration was
weak. Transient session ids would appear and disappear in the urls as
you moved from Postnuke to phpBB2. The mods we applied to phpBB2 were
very difficult to maintain through an upgrade.
Moving forward
General inertia kept us using Postnuke for several years during which
time it had a harder and harder time coping with the load. We forked
off a separate database solely for forum searches to keep them from
locking up the main database. We routinely applied security patches
and fixed little issues as they cropped up. Eventually we decided
that with the next generation of Gallery we should put out a whole new
website so we started looking around for alternatives.
When it came time to choose a new CMS, we wanted to make sure that we
chose one that scale to handle our needs, was under active
development, had a large and active community and was easy to
customize. We wanted to make sure that if we did have to make
changes, they could be mostly localized into the theme code so that we
could avoid tweaking the framework code and winding up with another
forked CMS. It had to be flexible enough that we could add in our own
apps like our paid support system, many of which are only visible to a
subset of our users or the development team. We gathered input from
as many sources as we can and eventually narrowed our choices down to
two four solutions: Xaraya,
Postnuke, Mambo and Drupal.
Xaraya looked promising, but they had not hit their 1.0 release yet
and we didn't want to be stuck in a bind if they were not ready when
we needed it. The latest version of Postnuke promised the easiest
upgrade path but among other reasons, we wanted to get away from
having embedded applications like phpBB2 for our main content. We
wanted very tight integration.
This left Mambo and Drupal and it was tough to decide between them.
They both seemed to offer similar capabilities and features. Both had
active development communities. Both had good scalability stories.
Neither had an embedded Gallery2 story at the time, but we were
reasonably confident that one would emerge over the months that it
would take us to port the site. Both had achieved a level of
stability and had a reasonable security story. As with all modern
CMS, both were modular and had flexible theming systems. Mambo seemed
to have a better forums system, but Drupal's seemed capable of
handling our needs and Drupal's node-based forums gave us greater
flexibility. It was Drupal's adoption by a few high profile sites
that demonstrated that it really could fulfill our needs. That
coupled with the fact that we personally knew several Drupal
developers who could help us with the migration made us choose Drupal
in the end.
The challenges
By far, the biggest challenge in porting was to move over all of our
data. Because the Gallery website contains a great deal of
information in the forums and has been around for a long time, there
are many internal pages and external sites that link deep into the
site. We wanted to be sure that we not only preserved all that data
but to save all the links also. It would have been very easy to
sacrifice these internal and external links or even start with a blank
slate, but it would have meant that we'd be severing ties with the
history of the project. There are still useful links out in the wild
that point to our old phpBB1_4 installation so we wanted to preserve
our history in one place, so we aimed to keep as much as we could.
We didn't have time to do this ourselves so we hired Kitt Hodsden on a contract basis to
do the data migration for us. Her goal was to figure out how to move
all of our data over to the new site in a way that would sacrifice as
little as possible. At first glance this did not seem like such a
difficult task but the details of the migration were extremely tricky
and demanding. In order for us to have a useful beta cycle, we
required that she automate the entire process so that at any given
time we could run the script and it would blast the beta site and
recreate a brand new one by importing all of the data from Postnuke
again. Our data is diverse, our links are many, Postnuke and Drupal
are dissimilar -- this was a complex task and it took Kitt many months
of effort to get it to a point where it was 80% migrated. This also
included writing several custom modules for us and migrating over the
data for them as well. A secondary goal of this project was to fund a
way for Kitt to create a migration path from Postnuke to Drupal that
she could then leverage for her consulting firm. There were many,
many niggling details to resolve but in the end we accomplished a
push-button migration script that let us get a beta up to find and
resolve all the remaining issues before going live.
We hit some snags
Partway through the process of porting to Drupal we started
reorganizing our documentation. In order to make sure that we were
using the right tools, we started experimenting with MediaWiki as our repository on a
second server. This turned
out to be a far better tool than the wiki implementation in Postnuke
or Drupal and it did not take us long to collectively decide to
migrate our data over there. This is when we discovered one of the
downsides to Drupal -- it does not have a good MediaWiki integration
story. By that point Kitt was halfway through porting to Drupal and
we were committed. But this left us in the unfortunate bind of having
two loosely linked websites without a clean integration.
During this process, James Walker got
the Drupal/Gallery2
integration module far enough that we could use it almost out of
the box on the new site. With a little work we added the features we
needed (like having the random image block in the sidebar) and sent
those changes back to James for inclusion in Drupal's CVS repository.
The Drupal forums module was not quite as good as we had hoped.
Drupal follows a post-and-comments model which makes some of the
nice forum moderation options like splitting certain posts off
of a forum topic into a new topic very difficult. The default
threaded comment presentation style did not mesh very nicely
with our forum ideal. For a site that is heavily forum based,
this was not promising. However, some investigation of the
contributed Drupal modules turned up code that could be massaged
into doing what we needed. It was not ideal, but provided
enough functionality for us to get by.
Making it look good!
Mike Harding's Logo
As Kitt started getting close to a point where she had a working
data migration, we started focusing on creating a new theme for
the site. We wanted a light and welcoming theme that would
provide our users with intuitive site navigation. To make this
happen, we hired Trae McComb to
design mockups and create a prototype of a Drupal theme for us.
He created the overall look of the site around Mike Harding's
winning submission in the 2003 Gallery Logo contest. Trae then
translated that into a PHPTemplate based Drupal theme based on
Lincoln's
Revenge. With some design help from Chad Kieffer, we put the finishing
touches on it to create a theme that gives us a warm fuzzy
feeling when we look at it.
We achieved our goal.. Now what?
Our goal was to launch the new site before we release Gallery
2.0 so that we kick off the new product with a whole new brand.
It took a tremendous amount of effort from several people but
we're very proud that we've made it happen. There are still
some outstanding issues we need to resolve, like figuring out
how to integrate our MediaWiki data into Drupal in a sensible
fashion and exploring other ways to improve our forums without
forking Drupal.
Our journey with Drupal is just starting. Now that we've got a
shiny new website we're going to take good care of it so that it
will last us for many years to come! We hope that you're as
happy with it as we are!