karniv0re's Journal: Big Trouble in Little Autonomy

Autonomy's IDOL server is causing us problems again. Probably not as many problems as HP, but problems nonetheless.

Firstly, we're running into INDEX QUEUE FULL exceptions possibly nightly. They're filling up our exception log database, and causing data to be lost. It seems to be happening between 8:30 PM and 9:30 PM. Weird.

While I was investigating that issue, I saw that the content/nodetable/ directory is filling up with 1GB entries that look like nodetable-nddX.db where X is an incremental number. Between 1-3 of these per day, which has caused us to eat through our 80 GB share in a month. At this rate, we've only got 9 days left, and I'll be on a boat to the Bahamas.

After talking to Autonomy support (who are probably super busy trolling Glassdoor), they suggested that our server is extremely fragmented. I tried running a DRESYNC, but it didn't do anything. Well, they suggested that that probably means the system didn't have enough resources to finish the task, which is probably true. The damn thing is using all the memory on the box and it's almost at 80% disk space. So what now?

Well, now the only option I have is to do an export to another database, DREINITIAL on the offending prod servers, and export the data back into them. I'm guessing it just knows to not fragment that shit or something, I don't know? We'll see tonight.

The shitty thing is, I was all raring and ready to do some solid dev work this week, and it has just not been my week. Operations keep hitting me with little bugs that I can't fix right now, but no less annoy me. It both annoys me that they bug me and that the bugs exist.

I made some solid progress on Vrapper last night, but time flies. Finally had some time to myself and my mother calls. This always happens. But instead of hitting the ignore button, I slid it to answer - which SHOULD NOT ANSWER THE CALL because Google Voice screens them. Well, apparently it did, and while I was waiting for the ringing to stop, I was talking out loud bitching to myself about how every time I get a night to myself, my mom or sister call me. I don't know how much of that rant she heard, but god I felt dumb. Note to self: stop making audible notes to yourself.

Now I'm trying to decide whether to hang out and do some development for Finance, or to head home and work on Vrapper until I have to come back tonight. Ehh, I suppose I can code for an hour. Should be able to get something done in that time.