At the risk of sounding biased* and even sycophantic, I’ve had a couple of cool experiences with StorageCraft‘s ShadowProtect software over the past few days that I figured I should share.  I’m not sure why I thought to share except that I guess humans just have this strange, innate desire to tell others about the cool things they do.  And I am, indeed, a human.

First let me set the proper stage.

The Back Story

(NOTE: I tend to be pretty verbose, so if you’re not interested in the back story, just skip down to The Point.)

There are 3 desktop computers enjoying ShadowProtection™ at my house.  Two are 32 bit Windows XP machines; the third is a 64 bit Windows 7 machine.  One of the XP machines belongs to my wife, while the other two machines are mine.  The Windows 7 machine is my primary computer, which I use for my work (when at home) and whatever casual computer-based pursuits I engage in, including media production.  I’m a closet filmmaker(ish), and the more powerful system is required to run all the production software I use.

I’m also a hobbyist musician and my home office doubles as a bit of a rudimentary recording studio.  Sadly, however, the recording software I use (Pro Tools 8) doesn’t run stably on 64 bit architecture and I haven’t been able to bring myself to pull the $500 trigger on the upgrade to 10, which is 64 bit ready.  PT8 is still great for my needs, so $500 is a tough pill to swallow even if it does mean that I have to keep my XP box healthy, if for no other reason than to be my primary sound machine (hey, isn’t that a band?  Oh wait…that’s Miami Sound Machine).

Being a bit of a geek (though not as much as you might think for a professional software engineer), I also have several servers at my house, rack and all, one of which I use as the backup destination for all of my instances of ShadowProtect.  It’s a Linux server with Samba shares mounted on each of the desktops.  I run ImageManager on my Windows 7 machine monitoring the backup destinations over the network.  I have them all on a pretty conservative backup schedule: weekly full backup with twice-daily incrementals seven days a week.  Nothing to see here.

My XP machine has been with me for a long time.  I can’t even remember when I built it (I build all my own computers; none of these pre-built monstrosities you buy at Wal-Mart or Best Buy).  All things considered, it’s been a great computer.   Very reliable.  It has been with me through several years of software consulting and contracting, through countless installs, uninstalls, reinstalls, compiles, de-compiles, etc.  Needless to say, after all this time and use, it has not been running at optimal performance for a while.  I’ve been slowly weaning myself of any dependency on it, copying files off, moving software as much as possible, etc. and last week I finally found myself in a confident enough state to take more extreme measures.  I wanted to blow the whole OS away, reformat and start fresh with pretty much only Pro Tools installed and dedicate the machine to just recording (a great gig for a computer heading into retirement).  Still, I didn’t want to lose anything, especially some priceless historical document (read “old journal entry, tax document, photo, etc.”) that I had forgotten was on that computer.

I should mention here that this machine also was the final resting place of files dating back nearly two decades in some cases, moved from previously retired computers, none of which I had sorted or cleaned up properly.  Hmm…I think I’m pack-ratting more than I’ve been willing to admit now that I mention it.  (In my defense, I still do occasionally find myself perusing those old files and digging up some gem that had completely escaped my memory.)

With ShadowProtect in place now, I was finally ready at least to consider the possibility of a full rebuild.

I should have done it sooner.

The Point

I’ve had ShadowProtect keeping backups of my machines for several months, but I wasn’t sure I fully trusted it enough to just yank the rug out from under my XP machine with the whole reformat-reinstall rigmarole.  Sure, I could just blow it away and if I found myself having forgotten to copy something important off, just virtual boot one of the backups and get it, but I’m a bit paranoid about that kind of thing.  So before I pulled the trigger, I wanted to be absolutely certain I had a valid backup.

My first validation was to run the Head-Start Restore (HSR) procedure from ImageManager to build a VMWare Virtual Machine (VMDK) file that I could launch.  The first couple of attempts were blue screen misfires of my own doing, but once I chose the right set of restore options (e.g. restore MBR, restore disk signature, host-independent restore, etc.), I was able to get a valid VMDK that booted up just fine.

Being a VM on my primary computer, though, wasn’t my ideal, I decided, so I dusted off some old hardware I had kicking around and decided to do the same restore, but from the SP Recovery Environment, directly to the hard drive of this old box.  Again, it worked flawlessly, even on that old hardware, which is slow and a bit painful to use (it’s not even a SATA hard drive; good ol’ IDE).  On that old hardware, the restore took about 7 hours.  Now I have a physical computer that’s an exact copy of my XP box as it was before I took any drastic steps.  With that kind of confidence, knowing I can get to those files in any one of a dozen ways now, I was ready to give my XP box a new life as a dedicated Pro Tools recording station.

The rebuild went off with only a minor hitch or two, mostly related to software (including ShadowProtect) that I needed to reinstall on the newly built box, but which I had failed to deactivate on the old install.  No problem, though: I simply booted up the copy box and deactivated.  (Try that with Mozy or Carbonite!)

At the end of rebuild day, I was feeling very satisfied that the computer was exactly how I wanted it.  All Windows updates installed, all software installed and configured, and it was running clean and smooth, just like new.  The last thing to do was to take a backup of the machine in its current perfect state (well…reasonably close to perfect anyway, as close as Windows gets).  I set up the schedule in ShadowProtect and opted to “execute now” and off it went as off I went to bed.

The next morning I remembered that when I was reactivating Pro Tools on the new install the night before, I had noticed that there were some updates that I hadn’t installed.  I was a bit too hasty to do so, though, because as it turns out, it wreaked all kinds of havoc.

<tangent length=”brief”>
See, my version of Pro Tools is a specific edition of the software that requires a specific piece of hardware as an instrument interface.  The hardware, called an MBox by Digidesign (now Avid), is a pretty simple USB peripheral desktop device with inputs on the back that support ¼ inch (guitar) and XLR (microphone) cables.  Without the MBox in place, plugged in to the computer, Pro Tools will not run.  My MBox is the original model, nearly 10 years old now (it originally shipped with Pro Tools 6 if that’s any indication).
</tangent>

In installing the update, I had neglected to read the “fine print” (which actually wasn’t so fine; I was just an idiot) that said that the original MBox was no longer supported.  The install worked just fine and it looked like some new cool stuff was going to be installed, but when I tried to run Pro Tools after that, the dreaded “can’t find valid hardware” error kept popping up and killing it.  So I uninstalled the update and tried to reinstall the original 8.0, followed by the one update I had previously installed, 8.0.1.  No luck, and after multiple tries of various uninstalls and reinstalls, it got to where Pro Tools wouldn’t even launch.  Not a good sign for my nice, clean, newly rebuilt recording computer.

Thankfully, the great Shadow came to my rescue again, rising like a towering evergreen to Protect me (ba dum bum) from the harsh rays of my own idiocy.

An hour later, after another restore from the previous night’s backup (this time without the HIR option), I was back to clean and fresh and ready to rock.

And all is well with the world.

*In the interest of full disclosure, I am currently employed as a software engineer for StorageCraft; however, I have had no hand whatsoever in creating or developing ShadowProtect, ImageManager, or any currently released StorageCraft offering.  I’m a relatively new employee and my work has yet to see the light of day in public (soon enough, though).  So I write this testimonial, as much as possible, as an unbiased user, not as an employee.

I’ll write more later.  (I know, famous last words, right?)

Kyle comes up with most of the musical concepts. He’s about as prolific a songwriter as I’ve ever met. So these are his babies. Zach and I just fill in the texture with other musical stylings to go along with Kyle’s original concepts.

The first tune, This Love, is our latest and probably most complete as far as song structure and direction, though it’s still far from finished as to production. That’s Kyle on the vocals (obviously) and rhythm guitar. I’m on the drums and bass. Zach hasn’t made an appearance on this tune yet.

Next is our least developed tune, Drop A. It’s just an instrumental jam at the moment based on some wild tuning experimentation. As the title suggests, the low E is dropped all the way to A. Kyle is on the acoustic rhythm. Zach on melodic lead guitar exploration. I’m on bass. Drums are actually a loop in this one, but I have a drum line that I’ll eventually lay down for it.

Finally, Utah High is also pretty near completion as to structure, but far from being production ready. Hey, it’s a demo. Kyle once again on vocals and acoustic rhythm. Zach’s on electric rhythm and lead. I’m on bass and drums.

Enjoy.

If we’re going to try to track our “quantum particles” (i.e. execution…see the series opener for explanation) through that nebula, the place to start is with the User Control Panel (ucp.php). That is, after all, where most battles take place, or at least where they begin.

So, from the top…

The script opens with several definitions and includes that contain more definitions. We’re not going to concern ourselves much with those. Just note that those lines appear in most of the public-facing scripts in phpBB.

In fact, we really don’t care about anything until we get to the session_begin() function. I went into some detail about this function in the previous post, so I won’t go into it again here. We’ll get into it later when we get back to the discussion of Overstock’s unique needs and how I was finally able to meet them.

After the session is opened, phpBB needs to detemine the permissions held by the user. That’s done by a call to the $auth->acl() function, which takes an array of user data as it’s argument. The user data array ($user->data) exists by virtue of the session that was previously opened for the user ($user). As that user object is constructed, user permissions are either queried from the database for that particular user (if a record exists) or they are queried from a “guest” account to allow the user to browse around the forums as a guest as allowed by the forum permissions (set by admin; outside the scope of this discussion).

Now that the session is created and the permissions set, the next thing we care about is $mode. You may have noticed a few lines up the line that defines the $mode variable by a call to the request_var function. I’ve never taken the time to read through this function, but my cursory observation and the obvious implication of the name lead me to deduce that this is phpBB’s wrapped method of extracting values from other places (such as REQUEST arrays). In this particular case, the value being extracted comes from the URL string, so it is a GET parameter called mode. If the user is not logged in, the value of this parameter will be login.

So, following through ucp.php, we come to a switch statement that is testing the value of $mode. For now, we only care about the block that executes if the value of $mode is “login.”

The original phpBB code checks to see if the user if registered. If so, the user is redirected to the index page; otherwise, the login_box() function is called, which (you guessed it) renders the phpBB login box. All well and good under normal circumstances, but in Overstock’s case, this wasn’t the desired behavior. As I mentioned in the first post, Overstock wanted to bypass the standard phpBB login process and replace it with their own “universal login” procedure. So rather than using the standard phpBB login screen, the desired behavior was for the system to check for an existing universal login session (certain cookie values), and if it existed, automatically log the user into phpBB; otherwise, redirect the user to Overstock’s univseral login screen, which would then send the user back to the referring page upon successful authentication. Once returned from the universal login, the user needed to be authenticated and logged into phpBB seamlessly.

Sounds simple enough; however there is one further complicating wrinkle that needed to be handled. In the event that the user had never used the Overstock forums before (i.e. didn’t have an existing phpBB account), the login procedure needed to create the phpBB account for them. So, effectively, the user would have two accounts–one universal login account for Overstock sites, and one phpBB account–that needed to be seamlessly synchronized. The phpBB account would be used for phpBB authentication and permissions, but the user’s experience has to act as though everything is controlled by the universal login account.

So back to the UCP.

To accomplish the desired effects, I took out all of phpBB’s original login code and replaced it with an if/else procedure that tests for a valid universal login session (ID). If found, the username and password for the user’s phpBB account are determined (I’m keeping the details of this procedure to myself for security reasons). Finally, it executes the $auth->login() function, passing in the username and password. If successful, it redirects to the index page and phpBB functions as normal from there with the user fully logged in.

So there it is. Mission accomplished…sort of. We’re now able to successfully authenticate a user in phpBB using an external authentication procedure. Of course, that assumes the user already has an existing phpBB account that corresponds to their universal login account.

But what if they don’t? How do we iron out that particular wrinkle?

Read on. We’re winning battles, but the war is still far from over.

Again, credit to Zend for making it possible to even follow that path, since without it, I would be left to a lot of guessing and just reading of code.

<tangent>Let me just interject at this point that this is one of the most annoying tendencies of Open Source software developers: they make the assumption that since they are complete geeks who would just as soon read code as a good novel, that everyone else in the development world should do the same. I think this is more arrogance than laziness or misunderstanding in many cases; it’s their way of saying, “if you’re not willing to read every line of code, you’re not a real developer like I am.” I can read code just as well as anyone, but I’m actually a successful professional developer, so my time is much better spent in worthy pursuits like actually building things and is too valuable to be wasted reading every line of someone else’s code to try to decode their mental dysfunction so I can understand how and why they have done things the way they have. Documentation, geeks!</tangent>

Theoretically, phpBB3 supports a modular authentication model, so by setting a particular value in the database that tells phpBB what module to use, and then writing the functions for that module, you’re able to integrate your own custom auth code into phpBB3 (I’m not sure if this was also available in previous versions of phpBB). In Overstock’s case, authentication had to be customized, so step number one was to indicate what the name of the authentication module is. This is done by making an entry into the phpbb_config table (assuming your table prefix is phpbb_). Normally this would be done through the phpBB admin control panel, but for the sake of really understanding the innards, I’m going to refer to the actual database table and column(s).

The table phpbb_config has three columns defined:

• config_name
• config_value
• is_dynamic

This structure is intended to allow for configuration flexibility. When you identify an authentication module using the admin control panel, a new record is entered into phpbb_config with the values assigned as follows:

• auth_method
• [the name of your auth module]
• 0

Once this is set, when phpBB authentication executes, rather than using one of its own auth modules, it will use the one named in this record. Sounds simple enough, but there is one important thing to remember:

No matter what name you assign to your authentication module, phpBB expects the file name to be prepended with auth_.

So, for example, if you specify that your authentication module name is custom, phpBB will expect that the authentication functions for your module are defined in a file called auth_custom.php (assuming the extension you’re using is .php). So, in order to use your custom functions, you have to create a file called auth_custom.php and store it in the includes/auth directory of your phpBB installation.

Get used to that convention, _custom, because it represents the pattern that will persist throughout the customization effort. (Of course custom is what I’m using for my examples; you’ll need to replace that with your authentication module name wherever appropriate).

Now that you have the file in place, you have to create functions that phpBB will recognize. Once again, the authentication module comes into play. To log a user in, phpBB calls its login function, which is a member (method) of the auth class. The login function is just a wrapper for the login function found in the authentication module being used. If you follow into the auth class login() function, the first thing that happens (after initializing globals) is it calls in the appropriate authentication module and tries to execute the login_ function for that module. So for our example, it will look for a function called login_custom() inside of includes/auth/auth_custom.php. Once execution enters this function, you’re in complete control and you can pretty much do whatever you want. Of course, it won’t likely do you any good unless you excute at least the most fundamental of phpBB’s login procedures…i.e. logging in.

To log a user in and take advantange of phpBB’s session management, you have to pass a valid username and passsword into the login function:

$auth->login($username, $password)

This will return an array of results whether it succeeds or fails. One of the array elements is called status, which is what you’ll use to determine success. PhpBB makes extensive use of constants (defined in includes/constants.php), and in the native authentication code, phpBB will compare the status element of the array with the LOGIN_SUCCESS constant value. If they match, login was successful; otherwise, it failed. From there flow control is up to you to handle according to your needs.

It should be noted that this whole procedure is started by a call to phpBB’s session_begin() function. Open up almost any of phpBB’s files (i.e. ucp.php) and you’ll see a call to this function near the top. In phpBB, the session class actually extends the user class (or maybe it’s vice-versa…I forget), which is why you’ll see this function called as a method of the $user object:

$user->session_begin();

If you follow through the session_begin() function (defined in includes/session.php), you’ll find the code that implements your custom authentication module (as of writing, this code begins on line 270); however, the real code we have to look at first comes before that, on line 231.

At this point in execution, the system enters an if statemement related to the session. Don’t ask me what all the different variables being compared actually are or where they are initially defined. With phpBB’s extensive use of constants, globals, and it’s own session management (rather than native PHP sessions), it’s very difficult to know exactly what is being compared to what and where any of it comes from. The long and short of it is this, though: if there is a valid session available for this user, enter this block of code; otherwise, it jumps down to line 334 and starts a new session.

With the session either restored or created, execution can now proceed.

We’re not even close to finished yet, but at least we know where to start. Battle won. Chalk one up for the good guys.

Sadly, however, the war continues…

I was hired to build their social media and community sites and tools (not alone, of course). In Overstock’s case, those “sites and tools” include blogs, forums, video sharing (a la YouTube), etc. I’ve had my fingers in basically all of these sites, but the big beast of late has been the forums.

Here’s the deal:

Overstock’s current forum system is built on phpBB2 and is very limited in it’s capability relative to the rest of the Overstock.com family of sites. The current project that I’m on has two primary objectives:

1. Migrate the Overstock forums to phpBB3
2. Integrate Overstock’s Unified Login system so that the forums authentication/authorization is based on enrollement in one of Overstock’s main sites (i.e. auctions, shopping, etc), rather than enrollment in Overstock’s forums.

Obviously the intent is to make the forums feel like a part of the Overstock.com family, rather than a third-party solution that has been crimped onto the corner of the Overstock sites.

So, why blog all this?

Simple: phpBB3′s code base is a freaking nightmare. So whether for my own future reference (never know when you’ll need it), or for some other poor geek that gets saddled with similar objectives, I figured I would document how I finally succeeded in making it all work. Obviously I’ll leave the specifics of Overstock’s unified login system out–that’s just ethics and good sense. But given that phpBB3 is open source (probably would be better referred to as “Obfuscated Source“) I am perfectly justified in divulging all I now know about its inner workings. Sadly, even after many epic battles, I still don’t understand much of how it all works. What’s worse, I think, is that I don’t understand why phpBB3 is built the way it is. I figure it’s for one of two reasons:

1. They have built their latest release on a horrible legacy code base with only enough modification to feel justified in calling it a new version. I could be wrong: they might have built it fresh from the ground up. If so, I’m even more disgusted by the phpBB development team, because that code is a mess. I find this hard to swallow, however, because the mere presence and pervasiveness of global variables throughout the code base tells me that the system design was lacking in foresight and flexibility, and was rather built to accomodate features that are making an unaltered journey from phpBB2 to phpBB3.
2. They have intentionally built on a hellishly cryptic architecture to keep it as proprietary as “open source” software is legally allowed to be. It’s certain that, once they’ve seen the code, developers and companies are going to think twice about building “their own” bulletin board systems by extending and modifying phpBB. The smart money would likely decide that they need to either stick with what phpBB offers (few modifications), or build their own system from scratch.

Of course, all of this could be completely unfounded. It could very well be that the code isn’t that hard to follow…once you know what to look for and where to look. But therein lies the biggest beast of all: trying to follow any logical flow through the phpBB3 code. In my exprience (and taking a page from the Simon Cowell Method of Introducing Similes), it was “quite like” tracking the movements of quantum particles at distances increasingly closer to the Planck Length. Every once in a while, with some heavy calculations and a fair amount of predictive luck, your observation might cross paths with the object you’re trying to observe; but mostly, the object just bounces seemingly instantaneously from one position to another without warning or trace.

At this point, I have to give props to the people at Zend. Were it not for Zend Studio (5.5) and their Zend Server Platform’s remote debugging capability, I’m certain I would have tossed in the towel weeks ago after determining that it was impossible to follow through the code manually. I was successful, ultimately, entirely thanks to the ability to step over and into specific parts of the code along the execution path; something that’s impossible to do in PHP without this Platform tool, as far as I know.

Let the battle begin.

After much research and countless balls hit in simulators, I narrowed it down to the Callaway, the Nike 1, and a Titleist (model I’ve forgotten). A little more effort eliminated the Titleist, and finally, when I managed to hit about 10 sim shots in a row with the Callaway that were dead straight and long (well…as much as such results can be trusted in a simulator), the Callaway won. I have never regretted it and thus began my commitment to Callaway golf gear.

I’ve gotten to a point now where I really don’t even need to test or compare anymore. I know Callaway’s gear will suit my game before I even swing the clubs, as was proven when my wife (isn’t she the best!?) bought me a full set of X-18 irons for Christmas two years ago. I didn’t even have to compare them to others, and by spring, I knew it was the right choice. I dropped about 5 strokes off my average that summer.

Next came my 56 degree X-Tour Chrome wedge, and finally, after a lot of credit card transactions on the Blue card, I earned enough points to get myself a 52 degree X-Tour Vintage wedge…excellent pieces of hardware that chipped another 3 or 4 strokes off my game last summer (pun intended, of course).

The final addition was the perfect ball. After playing many, and liking many, I finally found my magic with Callaway’s HX Tour and HX Tour 56 balls. They don’t last very long in my game; I tend to scratch them up pretty good with my wedges. But they have given me what I never have had in the previous 6 years or so of playing golf: spin control. They’re soft enough that even a hack like me with no real spin control can get the kind of spin that will stop the ball on the hardest of greens (as much as can be expected). Distance has never been a problem for me–I’m a pretty big dude–so for me, feel is key. Connecting with HX golf balls feels amazing, which is to say it doesn’t really feel at all. If I swing right, I can’t even feel the ball. That’s how I like it. Shave another couple of strokes.

So in a few short years I’ve gone from a complete hack who didn’t even dare register a handicap, to a proud 8.

I guess I can’t say for sure that the equipment is entirely responsible for my improvements; however, I’ve come to believe that there is one single element in golf that makes a greater difference in lowering your score than any other. It’s not the elusive repeatable inside-out swing, it’s not the putt stroke, it’s not the short game or the drive. It’s simply confidence. If you can strike the ball with confidence, no matter what shot you’re trying to make, your chances of success increase by orders of magnitude. For me, the feel of Callaway clubs gives me that confidence to know that if I do my job, the clubs will do the rest.

Now…where’s my sponsoship, Callaway!?