As of 3:26 PM EST on Wednesday February 1st 2012, Slashdot is now running completely on its new platform!
There were a few rough and long days in the last week completing the final capacity testing and roll out procedure, but it is now complete, and (*knocks on wood*) running smoothly. I owe many thanks to my team of engineers and the site operations team for all their help and support. The Editors here at Slashdot also deserve thanks for their help in testing, and their patience with me as I worked through the more complex pieces of functionality in the site. Though this project took a lot of work, I honestly believe that it was needed in order to allow us to keep Slashdot running, as well as to provide us the structural support we will need to build new features that will allow us to better serve our community into the future.

This is a test journal entry. I am only writing it to validate a test I am running on the system.
Move along, nothing to see here...

Well its been a very long week. Last weekend's move of our back-end database on to a modern MySQL version, and totally updated hardware went very smoothly. I know the editors, my team, and even some of our readers have noticed the improvement in site performance and stability. Awesome.
This weekend we were going to push our luck and try for two in a row, but we have decided to back off slightly, and delay the Apache 2.2 upgrade until the beginning of next week (probably Monday). During the course of the last week we were hitting all of our milestones in getting prepped for the launch, and things were looking good. However our first attempt at putting a fully load of traffic through the new system yesterday uncovered a few minor issues. Some late night coding and lots of caffeine got us ready for a second round of load testing the new setup today. While today's test did show marked improvement over yesterday's, there were still a few issues I was not able to resolve with sufficient confidence today. Another late night and more caffeine, and I could probably have everything ready to launch tomorrow, but I think it is wise to back up a bit, take our time, and be 100% sure that when we roll this launch out it will be rock solid.
On a fun note:
I just loved Spark Fun's new product video for today. I grew up in Boulder, and seeing the fun it looks like these guys get to have at work, if I ever move back to Colorado, that's where I want to work.
But working for Slashdot can be pretty fun too... I'm hoping the coming weeks will include a few episodes of Slash TV that I'll be helping out with... Going to check out The Maker Works here in Ann Arbor, and perhaps some fun with Lasers in the Slashdot/SourceForge parking lot in Dexter.

This is a journal entry testing our first release moving Slashdot on to modern versions of apache and ModPerl. Though this test instance is not available to the public yet, it will be comming soon... (this weekend provided all tests go smoothly between now and then).
Horray for modern software!

Slashdot is now running on upgrade database backend software, and new hardware. Anecdotal evidence suggests improved performance. Was anybody worried? I wasn't. Next week, updated front end web servers!

Sorry for the meta moment, but just wanted to call attention to the hard work my colleauge Vroom has been doing.
Hop over to his journal to check out some of the improvements coming out.

Update on ModPerl2, Apache2.2 port

Well I got through the problems with Apache2::Module::get_config. Aparently the object which comes as the first argument to the method for the custom config directive is not where the config parameters should be stored. Looking back at the documentation http://perl.apache.org/docs/2.0/api/Apache2/Module.html, I can see that it is clearly shown that $srv_config is being fetched via a call to get_config from within the custom directive subroutine, and that apparently is where the config should be stored... but it does seem kind of counter intuitive to me. But no use second guessing that now, at least I got it.

On a positive note: before I discovered my mistake with get_config, I thought perhaps it was getting mangled by Apache2::compat, so I spent much of the day finding and jumping through hoops to get everything to run without that. Though that effort didn't fix the problem I was chasing, it will be better in the long run to not need Apache2::compat, as that would drag down response times.

In my ongoing efforts to port slashdot onto modern revisions of Apache and ModPerl, I've made significant strides. This weekend I put in a few extras hours of work, just because it was on my mind.

Currently I am banging my head against Apache2::Module::get_config. I thought I had appropriately adjusted the code to use the new simplified mechanism for defining and handling custom apache config directives. Having added plenty of "print STDERR" debuging into my modules, I can see that the new directives are being read, and the associated sub routines to handle them are being called at server start time. However, calls to get_config from inside a request handler are always returning either 1 or undef. As far as I can tell from reading the documentation, it should be working. As I type this, it occurs to me that it could be getting fubared by Apache2::compat...

Off to test that theory!

As my colleagues are working to improve the comment system, I'm thinking about it. I had an idea today, and I'd love to hear what anybody reading my journal thinks about it:

Would it be useful to have a control that would either expand or collapse all of a comments ancestors, descendants, or siblings?

So, what do you think?

I am continuing forward with the re-platforming effort to move to Apache2.2 and ModPerl2. I have in the last few months be doing little bits here and there where I could squeeze it in between other projects, but today I got a good half day put into it. Before today I had updated the Makefile.PL's where appropriate to move away from Apache::ExtUtils, and instead use ModPerl::MM. I spent much of today finding all the CPAN modules that we rely on, and getting them built under the proper version of perl. After that I spent an hour or so tweaking config files where needed for the newer version of apache. Before heading home for the day I was actually to the point of trying to start Apache. It almost started, but I apparently missed something in updating the system to use Apache2::Module::add to handle the custom Apache config directives. But that will have to wait for tomorrow.

In comment system news: my colleagues are currently working on mock-ups for a few new controls to allow greater control over how many comments get loaded on an article page. Also trying to brainstorm around an easier/more intuitive way to handle setting thresholds for comment visibility (the current implementation is the "slider" widget with the two handles at the right side of the comment section header).

And there were, and it was good, for a while.

The comment system, and the discussions that take place in it, are a very important part of what makes slashdot what it is. I think it is a very powerful system, into which lots of time, thought, tests, trial and error, blood, sweat, unicorn meat, and tears have been put over the years. It is currently very complicated, somewhat fragmented and not well understood my a large number of readers. As I mentioned in a reply in one of my previous journal entries, the comment system is due to receive some much needed attention in the coming weeks. To that end, I am pleased to note that a release will be coming out today to address an issue we've found in the dispersal of moderation points. More than one or two readers have asked if there was something broken, or if they had somehow lost "moderation privilege", and I do believe that starting tomorrow, that issue should be resolved (thanks to the hard work of my friend/colleague vroom.)

Stay tuned for news of more love to come. (And feel free to let me know if you have special requests or concerns about the comment system)

Well the Hall of fame seems to be working for the few features it offered initially, and a few people have given me some good feedback and suggestions for features it could/should have. Once its been live as is for some period, I'll put my focus back on it, and hopefully implement some of the suggestions I've gotten.

In the mean time, I will be spending some time on back end work. While I may not be able to point to any "shinny new" stuff, I hope readers will see the results of this work in the form of stability, speed, and reliability. I am also looking for a contractor to help me out for a few months with the platform migration to Apache 2 and mod_perl 2... hence the title of this post. If you have a strong background with LAMP stack applications, and think you have what it takes to help us improve slashdot, I am accepting resume's. The position would be short term (2-3 months), but it is a chance to get into slashdot, and show us what you've got.

So, Can you ? (leave me a comment if interested and I'll get you contact info)

PerlJedi

P.S. there is also a full time position open for an entry level/associate engineer on my team. Go Here for details

Well, the new Hall of Fame is now live and open for business. I am trying to be patient, and to wait for it to live in the world for a bit before I jump into to look at the stats. Mostly I just hope that at least some people find it interesting, or helpful. I am moving into my next project next week, but will turn my attention back to the HoF from time to time, and hopefully add features. Please feel free to let me know if the new HoF is a disappointment, or if it is great, if you think it could be better if...

I am excited for my next project, though it will be a lot of work... Migrating Slashdot from Apache 1.3 and mod_perl 1 to Apache 2.2 and mod_perl 2. Though I have undertaken this sort of project of a few large systems, Slashdot will have some new puzzles to solve. I hope in the course of this project I will also be able to hammer out a few persistent bugs, and probably do some stream lining to get the page render times down a bit.

With the release of the new "Hall of Fame" comming later this week, I've squeezed one more feature into it for the first release. We will have a "Staff Favorites" section. Originally, I was thinking this would be only "Editors Favorites", but I thought (and my colleagues seem to agree) that it would be nice to show the "Favorites" of a larger portion of our team. That being said, I am not compelling any one (not even the editors) to share their "favorite" stories, but I will be giving the option to all of our engineers in addition to the editors. If that picks up, perhaps others will want to join in.

I wonder... if we offered the feature: would our readers like to be able to mark their own favorite stories? Maybe their favorite comments as well? This information could, if a user wanted it to, be displayed on their user page. Perhaps also, if we biuld out more of the system's "friend or foe" system, users could then browse their friends favorite stories/comments/submissions? (Must be honest: part of this htought is inspired by my friend/colleague Cogent's ideas.

Wow. My last journal entry really droned on to long. Feel free to tell me to shut up next time.

First round draft of the new "Periodic" Hall of Fame (as I've been calling it) should be showing up in the wild in this comming week. For the first release it is really just a sub set of the existing hall of fame, though broken down to smaller chunks of time. The really new piece here, however, will be the display of a the current users own stats next to each category. So next to "Most Active Submitters" a logged in user will see "My submissions", and next to "Most active Commenters" they will see "My comments". I hope by doing this people will be able to see that the new comers are not really that different from the grey hairs on a smaller time scale.

One other quick note: A feature we may be able to bring in V2 might be a visual representation of some of the data. For example perhaps next to "My comments" there may be a graph of # of comments a user has left per week in the last month. What do you think: Would that be cool? Usefull? Overkill? Please, share your thoughts...