Archive | April, 2013

Intermittent MongoDB [conn5] assertion 16550 not authorized for query Error After Upgrade to 2.4

So I recently upgraded the main MongoDB replica set from 2.2.x to 2.4.x for performance and mostly for security after the recent major MongoDB vulnerability.

Setting up the replica set in the first place wasn’t too troublesome, but maintaining has been a pain in the butt on and off, MongoDB does have some peculiarities. Once for PRIMARY member of the set crashed or hung or something, and the other 2 got confused, and rather than electing a new PRIMARY – just stopped working entirely.

Also the Ubuntu package doesn’t come with Logrotate capability, and the MongoDB log is very noisy, so I ran out of HDD space on a machine during one of my earlier installs.

Plus the default ‘security’ setup of MongoDB is a bit worring, it comes with auth turned off, no user/password login and it listens on all ports rather than binding to localhost by default. Of course all of these things are fairly easily fixed, but it makes me wonder how many poorly secured MongoDB instances there are out there on the public Internet.

Anyway back to this problem, at first I couldn’t upgrade to 2.4 because of a duplicate user error, after installed the new package it failed to start (rather than warning me during install) – so I had to roll back to the older version and lock the package there for a while. After getting rid of the duplicate users, I could upgrade.

But 50% of the time the app was erroring out with something like this:

Fri Apr 17 09:48:01.790 [conn5] assertion 16550 not authorized for query on databasename.collection ns:databasename.collection query:{ $query: { email: "abc@hotmail.com" }, $orderby: { _id: 1 } }

But half the time it worked, I tried all sorts of things, figuring out maybe the upgrade corrupted something. I totally deleted the data from both the SECONDARY MongoDB servers, and let them resync, updated the OS, rebooted, all kinds of things.

Even switched around the PRIMARY node, all of which..did nothing – I still kept getting the errors.

Finally after digging through the Ruby logs, which verified the app was getting an authentication error and digging through the database, I figured out what might be causing the problem. The Ruby logs were showing something like this:

Moped::Errors::QueryFailure: The operation: #Moped::Protocol::Query
@length=110
@request_id=93
@response_to=0
@op_code=2004
@flags=[:slave_ok]
@full_collection_name="databasename.collection"
@skip=0
@limit=1
@selector=redacted
@fields=nil
failed with error 16550: "not authorized for query on databasename.collection"

After some more digging, I found a user with the same username, in 2 different databases. So we have database databasename and database admin, both of which had the users databaseuser and admin.

You can search for users using these commands (you’ll have to auth first, if you set it up):

> use mydatabase
> db.system.users.find()

Do this on each database you have, to check for duplicate usernames, to list all databases just do:

> use admin
> show dbs

If you find any duplicate users, you can delete them with:

> db.system.users.remove({"user" : "USERNAME"});

So I deleted the admin user from databasename and deleted the databaseuser from the admin database, what I suspect was happening was 50 percent of the time it was authing with the right user and working fine, but 50% of the time it was authing with the wrong user (databaseuser in the admin database) and not having access to what it needed, and thus giving the assertion error.

After deleting the two users, and restarting the app on all the cluster servers – the error went away and I wasn’t getting intermittent app failures any more! So yah if you’re having this problem, just double check all the databases you have and make sure you don’t have any users in different databases with the same username.

When it’s all good you’ll see something like this instead of the assertion errors:

Fri Apr 17 10:37:16.197 [conn442] authenticate db: databasename { authenticate: 1, user: "databaseuser", nonce: "7c1c7s234r23fds", key: "9sdf907897sdf78979s8d" }
Fri Apr 17 10:37:32.725 [conn216] query database.collection query: { $query: { email: "abc@hotmail.com" }, $orderby: { _id: 1 } } ntoreturn:1 ntoskip:0 nscanned:1 scanAndOrder:1 keyUpdates:0 locks(micros) r:142323 nreturned:1 reslen:126 141ms

I could find absolutely nothing on Google about this and it seems like MongoDB 2.4 deals with users/auth issues quite differently from 2.2 – so if you are upgrading beware.

Tags: , , , , , , , , , , ,

Continue Reading 1 CommentInternet & Tech, Sys Admin & DevOps

Installing WordPress In A Sub-Directory Using nginx

So I had this issue recently, I had a WordPress site on the main domain, and another WordPress install for some member stuff in a subdirectory/folder inside the main install. The main site worked fine, and the admin panel etc. But when you tried to change the Permalinks, everything got a 404 error.

I noticed that all requests were being routed back to the main domain (without the additional folder) so I guessed it was probably something to do with nginx, most likely this:

location / {
# This is cool because no php is touched for static content
try_files $uri $uri/ /index.php?$query_string;
}

Which broke the rewrites inside the folder and sent everything to the main site, to get it working I had to add a couple more location directives for the sub-directory – now all different kinds of Permalink configs work.

This got the site working just fine:

location /subinstall {
root /home/maindomain.com/public_html/subinstall;
index index.php index.html index.htm;
try_files $uri $uri/ @wp;
}

location @wp {
rewrite ^/subinstall(.*) /subinstall/index.php?q=$1;
}

With /subinstall being the name of the sub-folder with the 2nd WordPress install.

Tags: , , , , , , ,

Continue Reading No CommentsInternet & Tech, Sys Admin & DevOps

sysstat – sar Invalid system activity file: /var/log/sysstat/sa19

After a recent upgrade (not sure which) the sar command stopped working, with the error output:

Invalid system activity file: /var/log/sysstat/sa19

The saXX can be any number, and relates to the log file – which for some reason is corrupt/missing/unreadable. In my case it seems to exist, but doesn’t function.

I’ve found the best remedy to this to be purging sysstat from the system and reinstalling it, fairly trivial to script for multiple machines and seems to have fixed it up just fine.

Here’s the command to run to fix it:

sudo aptitude update;
sudo aptitude purge sysstat -y;
sudo aptitude install sysstat -y;
sudo sed -i "s/ENABLED="false"/ENABLED="true"/" /etc/default/sysstat;
sudo /etc/init.d/sysstat start
/

The lines from top to bottom update the repo list, remove sysstat and purge it’s settings, install sysstat, set the sysstat config file to true so it runs, then start the sysstat daemon.

It should be rocking again after that 馃檪

root@server1:/var/log/sysstat# sar
Linux 3.8.4-x86_64 (server1) 04/19/2013 _x86_64_ (8 CPU)

08:59:15 AM LINUX RESTART

Tags: , , , , , , ,

Continue Reading No CommentsInternet & Tech, Sys Admin & DevOps

Heineken UEFA Champions League Ibiza Final!

So it’s that time of year again, time for the UEFA Champions League! For those who are not familiar, the UEFA Champions League is the most prestigious club platform for international football stars and their clubs. With over 150 million TV viewers watching live coverage of the UEFA Champions League in 220 countries and territories every match week.

Heineken is the proud sponsor of UEFA Champions League, this year Heineken is sending some of you lucky guys over to Ibiza to watch the all-inclusive VIP Screening of the UEFA Champions League Final!

Ibiza is one of the hottest party capitals on the entire planet, everyone’s dream in the UK is to head to Ibiza in the summer during their teenage years and party it up!

Ibiza

I’d personally love to head over to Ibiza, perhaps hit Amensia or Pacha, famous names from my youth. I’d be interested to see what they are like now!

Cocoon at Amnesia - Ibiza

Plus they have some amazing foam parties over there! Imagine that, looks like fun no? They truly have a spectacular party scene over there.

Ibiza Foam Party

And the beach over there is fantastic, especially at sunset – the whole Cafe del Mar chill-out music scene came from Ibiza (from a bar there of the same name). It’s a gorgeous place, even if you aren’t that into clubbing.

Ibiza Beaches

In order to have a chance to win – and get yourself to Ibiza, you need to participate in the Facebook App contest, you can check out the app here:

Kick-Off To Ibiza

Heineken Kick Off To Ibiza

Just like the page to get started!

The more Tweets & Facebook Check-Ins created, the sooner the location to the hidden tickets will be unveiled. Once it鈥檚 unveiled, consumers will stand a chance to be invited to the final on-groundchallenge where they鈥檒l use their wits to compete for the all-inclusive VIP experience in Ibiza

Heineken Kick Off To Ibiza

There’s none of the teams I really follow still in the UEFA Champions League last 16, but I do enjoy to watch the football of Real Madrid and Barcelona. So I shall be watching a few games and having a few Heinekens! Plus having a go at the game to see if I get myself to Ibiza!

The contest period is from April 2nd until April 20th.

Give it a shot, who doesn’t love football and partying – Kick-Off To Ibiza

Tags: , , , , , , , , ,

Continue Reading No CommentsAdvertorial

Went To Bali For My 35th Birthday!

So it turns out for my 35th birthday, my wifey took me to Bali! Never been there before and always wanted to go.

Pretty interesting place, although next time I’d probably choose to stay in Seminyank or Ubud rather than Kuta, it was good to see the place though. Food there is pretty awesome, it was spectacularly hot when we went though.

Got to try a few local craft beers, but I couldn’t find Storm anywhere, ended up with some Stark though – which was decent.

Tanah Lot Sunset
This is the sunset from the famous temple Tanah Lot.

I’d definitely like to go again and spend some time exploring the place further, plus of course I really want to go diving there 馃檪

So many other places to go though, arghhh – need more time and more money!

Tags: , , , , , , , ,

Continue Reading 1 CommentIndonesia, Trips & Travel