karniv0re's Journal: IT Heroes and Big Data

Many moons ago (circa 2006), when I was a young lass studying Computer Science, I had a teacher that really stood out. He swore freely in class, wasn't afraid to call a kid stupid for asking stupid questions or not being able to read code, and regaled us with probably one too many fantastic stories about how he played football for the Chiefs. And while he could be an asshole at times, I probably learned more from him than just about any other throughout my college years.

He didn't just teach C/C++, but lots of other practical tidbits along the way, such as Bash command completion, exiting a shell with CTRL+D and why that works (signals end of input), using Splint, and a myriad of other little things that where maybe outside the scope of the class, but added a lot to my IT tool belt. This made him one of my first IT heroes.

Of course there are others that I have never met and are well known throughout the community - Bruce Schneier, Theo DeRadt, Ken Thompson, Donald Knuth, etc. Those guys have all earned their place at the top of my IT heroes list.

So how does one become an IT hero? It seems there are two paths. One can teach, or one can code their way to fame. I'm not sure that I'll ever be anyone's IT hero, unless I get a cool intern someday who is eager to sponge up knowledge. So far, most of the interns I've been around don't remind me of me (i.e. they didn't come in with a breadth of knowledge, and they don't seem super interested in tackling big projects to expand their abilities). Maybe someday...

On a completely different note, I am loving this Data Science class I'm taking on Coursera: https://class.coursera.org/datasci-002/lecture

And I'm also a little ashamed that it's taken me this long to learn about MapReduce and NoSQL. Really ashamed, actually. There are so many problems I could have solved with these tools had I known about them years ago. Of course, we at the big corporation have only recently begun to embrace Hadoop, while NoSQL has not yet entered the enterprise at any scale.

And with my current project, I'm wondering if there's a place for it. I'm writing a Hypermedia API and I want to track requests, responses, failures, etc for analytics. In our existing internal RESTful services, we're storing entire JSON objects in Oracle CLOBs. This has of course started to bog down Oracle and chew away at disk space. And it's a bitch to even try munging for analytics. In fact, SAS is doing a PoC on that data, so we had to parse it for them and put it into a table. So that whole thing sucks.

So I'm like, 95% sure I don't want to go that route, especially since that database is already having growing pains. But I want to capture that data. But (yet again, "but) I am just learning about NoSQL and Hadoop, so I don't know if I should try and incorporate one of those systems into my design, since I know my first implementation is not going to be great. There is the option of writing to flat files and saving those off somewhere for later parsing, but I know that once I get this thing set up, I'm probably not going to get a second chance to implement this. A new project will come along and this will be forgotten about until something breaks or someone asks me for stats.

I'm currently on the NoSQL chapter in the Data Science class, which is basically a 50k ft. view, but it's more than I've learned about NoSQL up to this point. Again, I'm a little ashamed that I haven't dedicated some time to it up until now. Better late than never though.

We shall see.