Progress Without Perfection

Initially, I thought that given the nice detail in the Gravity team's papers, I would be able to implement these and get very close scores. It has turned out not to be the case. As the models get bigger and more complex my results have moved farther away from theirs: not terribly far, but far enough. This could be a factor of details they didn't include in the papers, details I missed in the papers, or my misinterpreatations (or programming errors) in implementing them.

This was irritating me last week, because I didn't really want to move on to the next level of the algorithm until I had gotten the previous one working. But now I've given up on that. Even though I am not as close to their results as I would like, I am making progress and now with my implementation of their 250 feature, retrained model BRISMF#250UM I have a quiz score of 0.8967.

The idea behind the retraining is that as the model adjusts features, the users at the end of the list are being adjusted against different movie features as the top of the list. This is because everything is trained simultaneously. The retraining as initially conceived by the Gravity team, was to reset the user features and lock in the move features. Thus, when training again users are adjusted against fixed movie features, hopefully for better results. But they ended up finding better results by not just retraining the now reset user features, they also continue to train the movie features from the original run. This ended up with much better results, and is how I got the 0.8967.

What I was doing today was adding in the neighborhood based correction to the predictions. I spent a long time looking at the formula, because I didn't really get it, but I think this is what the idea is.

In general, a neighborhood based (NB) approach uses the ratings of other users to determine what you would rate something. Since some users are more like you than others, contributions are weighted depending upon the similarity between the 2 of you, which can be determined in various ways.

In specific to Gravity's NB correction, some portion of the predicted error is added on to the regular MF prediction. The predicted error is found via a NB algorithm. Essentially you look in the training set (the ratings you have the answers for) and find what the error of each other rating by that user is. These are combined based on similarity weights again, to make similar items (in the case of Gravity's paper) contribute more to the projected error.

I'm still working on how to determine the parameters for the correction, but it looks like it could be promising.

Don't Rush Your Math

I've been trying to implement Gravity's BRISMF#250 model but have been getting much higher RMSEs.  I had no idea what was wrong so I decided to move on and start implementing the code to retrain features after the initial training.  To test I had to rerun my BRISMF#1 model which resulted in an RMSE 0.01 higher, which made no sense.  I looked through my code for changes I've made since I initially ran that and came across an error.

Most algorithms will require different ranges of random values to fill the feature arrays.  But, the way Java's random double generator is set up, it will only return a value between 0.0 and 1.0. Since that's not going to be helpful, I made a quick equation to modify the range. Unfortunately I rushed through what should have been a simple equation and came up with the following.

value = randomDouble * (max - min) - min;

The problem with this is if you wanted a range between -0.01 and 0.01, you end up with one between 0.02 and 0.03.  It should have been:

value = randomDouble * (max - min) + min;

Now that I've fixed this I can try BRISMF#250 again.  Lesson learned: don't rush your math (even really simple math).

Sucess (Relatively)

So now that the data is sorted by date as the Gravity team describes in their paper everything worked according to plan.  It ran in 13 epochs instead of 50-ish and got a .9217 RMSE instead of 1.0014.

Now I can finally start playing around with the other things they've tried.  Maybe at some point I can try something of my own.

Matrix Factorization

The idea behind matrix factorization (which is similar if not the same as what the Netflix community is calling SVD, singular vector decomposition or something) is that you can estimate an I x J matrix R (the ratings matrix) by multiplying two smaller matrices: an I x K and a K x J.  Each of those K rows/columns is known as a feature and the matrix factorization (MF) algorithm will estimate the scores for each user and movie.

What this allows you to do is essentially generate scores for how much action, comedy, romance, etc. is in a movie.  And also how much a certain user likes those aspects.

So I am using the papers written by the Gravity team to try this out.  They've outlined it really well, down to actually giving you the algorithm in psuedocode and listing different parameters they've tried.

So I started working on this and eventually got it to run.  But the algorithm that they say takes 13 epochs (loop cycles, essentially), took mine about 50 with the same parameters.  So now I'm trying to resort the data by date as they have to see if that will work.  But since that took me all day yesterday I decided to submit a prediction from the features I had built before.  It got a 1.0041 RMSE on the quiz set, which is not good.  

Just to give you an idea: Cinematch (Netflix's algorithm) gets a .9514, and the leader (Pragmatic Theory, as of today) has a .8597.