Become a fan of Slashdot on Facebook


Forgot your password?
User Journal

Journal Journal: The Streaming Data of My Consciousness

First of all, why is the "Write in Journal" button at the goddamn nether reaches of Slashdot? Are you trying to get me to not write my thoughts here? Too fucking bad. I've been doing it for over a decade, so deal with it.

This post will be largely drunken and angry and in a stream of consciousness style because I am drinking, angry, and don't care. I mean, most of my posts are stream-o-consciousness posts, but I am not always drunk. I'm actually not drunk... yet. I'm working on it. Drinking a LeatherheadRed which is quite delicious, and I took a shot of Evan Williams right when I got home because, well, that's where I'm at.

The Lady asked for more money to buy school books today. I don't think it would bother me as much if she just said, "heyyyy, I might need $500 for books. Can you swing that?" Rather than, "So I might need money for books." Me: "How much." Her: "I just hate how they do this. I hate not contributing. blah blah blah." None of that helps. In fact, it just makes me angrier. Just tell me what you need so I can figure it out. Right now, I'm still living on 401(k) loan money. The stupidest loan I've taken out, but what choice did I have. It is keeping me off of credit cards, but is in effect, decreasing my retirement significantly, and tying me to the company that I would like to have the option to leave.

This goddamn company, man. Stock is dropping. They're predicting another shitty year in 2016. For FUCKS SAKE. THIS IS NOT THE TIME. Everything else seems like it's falling apart. And what bothers me probably most right now is that I'm having trouble on my last homework in Probability Models. We're doing Renewal Theory, and shit got hard. All out of nowhere, everything was like, "Hey! We heard you've been doing well! We're here to fuck that up for you!" I have to turn it in tomorrow. I could turn it in and take a hit on it, which would keep me at a low A, high B, but for one thing, I want the A. And another, at least one of the problems on this homework is on our final, which is worth a shitton, so I HAVE to understand it regardless. So that, on top of realizing I spent over $600 on chameleons that we had to take to the Humane Society because the cats kept fucking with them.

Just came back from a break. I checked Facebook. That was a mistake. Read some bullshit about Nassim Taleb hating on GMOs. Goddamnit, nobody is good. Everyone has a part of them I will hate. Maybe that's just life. I'm sure there's things you'd hate about me (least of all my writing). I guess that just gives me more of a stance to take while reading Antifragile. The book overall is quite good and inspiring, but some of it must be taken with a grain of salt, or from a certain context. Like when he says that nothing good comes from universities, but rather from tinkering and hobbyists. Well, that's true for the big things (think the automobile, Microsoft, Facebook, etc), it's not true in general. But that's not what Taleb deals in. He deals in Black Swans - unforeseen groundbreaking events - good and bad, in which case, scientists being wrong about GMOs (lots and lots of scientists... lots) would be a Black Swan. Still, he puts the burden of proof on the scientists to prove that he is wrong. And that is not good skepticism or science. If you're bringing the skepticism, you better have science on your side. Unfortunately for Taleb, it is not. Can't get behind him on that one. Also, he resorts to name calling. That's pretty sad for a well-respected best selling author and academic. :/

Well anyway. My sister is badgering me to do her statistics homework. I partially want to to see if I can do it without remembering much from Prob & Stats I, but I also know I have a lot of shit to do and she's not learning anything if I do it for her. So I told her I'd give her an hour for questions. I have my own final to do! I'm genuinely stressed about this now. I want to do well. I want an A. For the first time in my academic career, an A matters to me. Demonstrable understanding of a topic matters to me. I want to be able to say, Yes, I took Probability Models. I can show you how to do Renewal Theory.

Alright, I'm gonna watch some 'flix and hit the sack. Peace. Hope you enjoyed the stream.

User Journal

Journal Journal: As the Path is Revealed, the Map Gets Bigger

I've been on the Data Science road for about 5 months. I initially became intrigued by the idea of Data Science on January 5th, 2015. This came about when I inquired about starting my master's degree in mathematics and was informed about a concentration in Data Science. "What is Data Science?" I wondered. So I started looking into it.

Here's my initial thought progression over time:

"I have no idea what that is"

"Sure looks trendy!"

"I bet this is one of those things everybody and their dog will want to do but without having the math chops to really be good... just like developers."

"Ooohhhhhh, look at all these tools! Hey! I've heard of a lot of these! Hadoop! CouchDB! MongoDB! Ummm, Spark? Dremmel? Spanner? Voldemort?! Where does this list end?!?!?! This must be Data Science!"

"Oh, so Data Science is an umbrella term. Underneath that is Predictive Analytics, Machine Learning, Data Engineering, Data Architecture, Computer Vision, Natural Language Processing [list goes on]. Ok, so we're back to Computer Science."

Me now: "Why don't they just call it Data-centric Computer Science?" "Because it's not catchy enough and it wouldn't pay as well." Oh yeah.

So we've come full circle. It's always been Computer Science. Some of us just took more math classes. All of those tools I mentioned in my post Dear Gournal have as much to do with data science as Matlab has to do with Mathematics. You wouldn't say Matlab is Mathematics. You would say Matlab is a Mathematical tool. In the same way, all of those technologies are Data Science tools, but they are not Data Science. I'm glad I'm realizing that now.

I'm almost through with my Probability Models class. I am somehow riding on a low A, and hope to finish strong, but the latest lessons on Poisson Processes and Renewal Theory are clouding my head. Still, it's been a very good class. Now I know what actuaries do! And I know I do not want to be one. Still, probability will never not be useful. It is at the core of what I want to do. AI, for example, is heavily based on prob. DS which is heavily stats based, is inherently prob based as well. Predictive Analytics, for example, would be impossible without probability theory. I hope to bring some of that to the table. Next semester, I'm taking Prob & Stat II. I think this class prepared me pretty well for it.

But while I now realize what Data Science is more or less, it only makes me realize how much I don't know. It's not as simple as learning a few tools and technologies. It's about learning the fundamentals of statistical analysis and probability, and then the things that build on that, like machine learning and predictive analytics. I'm excited and scared at the same time. It's terrifying if you try to eat the elephant in one bite. So I'm trying to take it a byte at a time, starting with the toe.

In my spare time, I'm working on a sports database, and a statistical analysis of a fantasy prognosticator. I've finally got the database together, and now I'm working toward the guru analysis. It's taken a little over a month, and I'm guessing will take another month to finish up, but I'm proud of it nonetheless. It's stupid but has been a fun exercise in Data Science.

Not much has changed at work except that I now have a project, finally. I am tasked with bringing Instant Messaging to the company. I mean, we already have IM, but it's internal only. This has to be internal AND external. So I'm rolling with XMPP. At least I hope. I have a meeting with our security team tomorrow to discuss the feasibility (security-wise, not technical). I think it's going to be a fun project. I'm planning on rolling with OpenFire. Should be pretty straightforward.

Anyway, that's it for now. Gonna go read more Anti-Fragile.

User Journal

Journal Journal: Year of the Black Swan

I'm reading Anti-Fragile, a book by Nassim Taleb, who also wrote a book called the Black Swan. I haven't read that one, and I'm thinking I should now, although he talks a lot about Black Swans in Anti-Fragile. A Black Swan is a disaster of levels previously unseen. Things like the 2008 financial meltdown, the Fukushima nuclear disaster, or Hurricane Katrina. In most Black Swan cases, people are only prepared for what they have seen, or what lies along the mean of the normal distribution. They don't plan on outliers, even though outliers are what destroy lives.

I thought 2013 was a bad year, because I couldn't do a lot of the things I wanted to do because I had six weddings to be in, which essentially drained my vacation time and my bank account. I am a major opponent of weddings because of that year. Fuck your wedding. But more on topic, I didn't think 2014 could be that bad, but knew that if it was, I could endure it. But I made the mistake in the assumption that 2013 was the worst it could get. 2014 proved to be even worse due in large part to my girlfriend moving in and bringing her horrible attitude at the time with her (that was due almost entirely to her job situation). It was a year fraught with sickness, strife over a troublesome dog that she did not want to let go of, an apartment complex under construction for 7 months, complete with an unbearable drilling and jackhammering into the walls that made sleeping impossible for her, a few near-breakups, and more sickness.

And then when things started to clear up - she got into anesthesia school, we got rid of the dog, they finally finished construction - another Black Swan. She was diagnosed with Meniere's Disease. Something that could be devastating for not only her career but her life. We've been living with it, and managing it, and she got into school. But as the Black Swan principle dictates, the worst is yet to come.

Now she is failing school. If she can't find a way to pull her grades up, the $100,000+ gamble will have gone to the house and I possibility of getting out of debt in a swift few years on good anesthesia pay will have been but a fantasy, and I'll be paying off debt well into my 40s.

This is a lesson I will take with me for the rest of my life. The Black Swan is very real. And the fragility that I have subjected myself to, has made everything so much worse. I took out a $25k Upstart loan last year. I'm already through that money. So I took out a $20k 401(k) loan. That made things even worse. I can't quit my job (or lose it!) because I will owe that money immediately back. On top of it, I've stopped contributing to my retirement fund. Again, fragile.

I'm in such a bad position, and it is mostly because of this high stakes gamble. If she can't pull through, then we are more or less fucked. I still don't know what to do. I don't really have a choice anyway. I am trapped in this fragile nightmare, at the whims of the Black Swan. Yesterday, she got a nail in her tire, which I took into the repair shop, and of course, they didn't have her size. So, to avoid her having to ride with her classmates (she can't stand being around them), I rented her a car. I don't love spending that money, which essentially doubled the cost of the tire, but more so, I didn't opt for the insurance or adding a driver. My Black Swan mind scolded me the entire drive home. A wrecked car, with her driving would be the biggest disaster yet, which is the definition of a Black Swan. Yet I didn't stop to correct it before driving away. And this is how the Black Swan thrives. It preys on those who are not robust or anti-fragile. And so, until I return it hopefully later today, I have to hope that that gamble does not result in a disaster.

And then I have to worry about the bigger impending Black Swan. The fact that she has to get a 91% on her final in order to pass her anesthesia class. We are probably just fucked.

User Journal

Journal Journal: Sick of Being With the Sick

This won't be a very tech-y post. I just need to vent. I thought, maybe, JUST MAYBE, the end of 2014 would be the end of all of our problems. But then 2015 comes along and it turns out, it is trying to one-up 2014. I'm talking about illness. My girlfriend has somehow, since 2014, went from being completely healthy, to being one of the most fragile people I have ever known.

Let's break down the numbers.

2014-01-02: Gastroenteritis - ER visit
2014-03-02: Tamarind allergy discovered - ER visit
2014-05-11: Sick (cold or something)
2014-06-01: Sick (cold or something)
2014-08-21: Tamarind allergy from Mexican food - ER visit
2014-08-22: Bad reaction to steroids from Tamarind allergy - ER visit
2014-08-24: Still having reactions from the steroids, mistook it for thyroid problems - ER visit
2014-11-20: Began having Meniere's episodes, though didn't know what it was
2014-11-21: Began having Meniere's episodes, though didn't know what it was
2014-11-22: Began having Meniere's episodes, though didn't know what it was
2014-11-23: Began having Meniere's episodes, though didn't know what it was
2014-11-24: Began having Meniere's episodes, though didn't know what it was
2014-11-25: Began having Meniere's episodes, though didn't know what it was
2014-11-26: Began having Meniere's episodes, though didn't know what it was
2014-11-27: Began having Meniere's episodes, though didn't know what it was
2014-11-28: Began having Meniere's episodes, though didn't know what it was
2014-11-29: Began having Meniere's episodes, though didn't know what it was
2015-02-12: Meniere's episode
2015-04-11: Meniere's episode
2015-05-08: TMJ
2015-05-15: Swollen lymph nodes
2015-05-16: Swollen lymph nodes
2015-05-17: Swollen lymph nodes
2015-05-18: Swollen lymph nodes - ER visit
2015-06-01: Cold
2015-06-11: Meniere's episode
2015-06-25: Meniere's episode
2015-07-05: Meniere's episode
2015-07-09: Meniere's episode
2015-07-18: Meniere's episode and stomach ache
2015-07-19: Stomach ache
2015-07-26: Meniere's episode
2015-07-31: Meniere's episode
2015-08-02: Meniere's episode
2015-08-03: Meniere's episode
2015-08-09: Meniere's episode
2015-08-12: Meniere's episode
2015-08-14: Meniere's episode
2015-08-18: Meniere's episode and cold
2015-09-11: Meniere's episode
2015-09-12: Meniere's episode
2015-09-13: Meniere's episode and stomach ache
2015-09-14: Stomach ache - ER visit
2015-09-20: Meniere's episode
2015-10-06: Meniere's episode
2015-10-11: Stomach ache
2015-10-12: Stomach ache
2015-10-13: Stomach ache
2015-10-14: Stomach ache
2015-10-15: Stomach ache

Those are just the days I documented. There are countless others where she just didn't feel good because of Meniere's or whatever, and times she didn't even tell me she wasn't feeling good. But from just those days alone, since January 1st, 2014, she has been ill 7.5% of the time (see, at least I'm including a little data science in here). That is a horrible batting average. That's averaging out to be almost a month of illness per year. I'd be stoked if I got that much vacation time! And it is terrible quality of life. And just when she was adjusting to the Meniere's and starting to figure out how to somewhat control it, she gets hit with this GI thing. Of course the doctor's don't want to even try to figure out what it might be. They just mumble something about IBS and walk out. Fuck our health care system. I think we should start calling it a "health don't-care" system. They only things they seem to be concerned with is passing out antibiotics, operating on people for things that could have been prevented in the first place, and fixing erectile dysfunction, which, God knows, is the world's biggest problem. For fuck's sake.

The stress this brings on me is causing me health problems too. My hair has thinned out a lot in the last 2 years (though, some of that can be blamed on my line of work), and a gigantic cold sore just reared its ugly head. And lord only knows what I'm doing to my liver to cope with the stress. I don't know what I need. I briefly fantasized about buying a one-way plane ticket to the Virgin Islands and just disappearing. Why there? I dunno, first place I thought of that was far away and didn't need a Visa. Maybe I could be homeless for a while. Start with some manual labor, working my way up to owning my own business, and bang lots of tourists. And then the cold reality of my life slaps that fantasy out of my head. Plus, it would just make her life harder, and I would never want that. She deserves so much better than what she is getting.

I don't know what to do at this point. I'm running out of positive affirmations. For myself included.

Sorry for the personal garbage. I thought it might feel good to just get out in words, but I'm not sure if it is helping or just confirming what I already thought. Anyway. Off to a stupid meeting where more people will probably just piss me off.

User Journal

Journal Journal: I Hope the World Burns

There's a scene from Nickelodeon's Salute Your Shorts I remember where Donkeylips and Spud were assigned to trash pickup duty and Donkeylips wasn't going to make weight to go to a wrestling tournament or something, and it all boiled down to him probably not getting to go eat some delicious lobster. I don't know why I remember this so vividly, but I guess it has always stuck in the back of my mind. In that scene, the trash bag rips open and all the trash they had collected starts blowing all over and Donkeylips says, "Garbage! My life is garbage!"

I feel like that lately.

This entire year has been nothing but rejections and watching my career sink. It started with not getting the architect job. I have since considered this blessing in disguise (not the paycheck part), as the action of that rejection caused an equal but opposite reaction in the opposite direction of Data Science. Now, while it is cool that I was spurred in that direction, there's literally nothing here helping me get there.

Earlier this year I felt like I needed a breadwinner of an idea. Something to cement my bonus. That idea was the automation of one of our accessorial billing line items, which I estimated could have brought in around $600k per year. I caught the ear of the guy who could have set that project in motion and then... nothing. I don't know what happened. If there was fear or trepidation in pulling a (slightly) bold move. But for whatever reason, the project never got initiated.

With that out the window, I was given the task of integrating systems with our partners over at R (names changed to protect the guilty). Since they suck at thier jobs and life, I knew this probably wasn't going to actually happen anytime soon, but I needed something to make the project pop. So I thought I would build out our entire portal as a B2B Hypermedia REST API. That is pretty much done and it is just sitting there waiting to move forward, and what's worse, I don't think anyone even cares. No one. I haven't advertised it yet, because nobody even seems to care that I'm literally doing nothing.

Today, I got my mind really hooked on this problem of predictive analysis. Since I'm in a Probability Models class, I successfully modeled one of our processes as a Poisson Process. I know that Asshat is already working ("working") on this with our Data Science team, but they're going to take supposedly months to get some proprietary 3rd party system up to do forecasting, and I know that I can have a crude but fairly effective model up in a week or less. So I took it to the big boss. But my pitch must have been off, because he just kinda said, "Cool!" and dismissed it like it was a neat little chart. FUCK. What do I have to do around here to get some important, thoughtful work?! Who's dick do I have to suck?!

So here we are, on the brink of the 4th quarter and I have nothing to show for this year. Unbelievable.

I literally don't even need to come in. I could have stayed home for 90% of this year. That's how far I've advanced my career in 2015.

My influence here is fading too. Our new architect doesn't clue me in on anything he's doing (but has made good friends with the other developers), my boss was let go, so now power-hungry Asshat has all the cool projects, my new boss doesn't really have a nose for strategic development, since his specialty is in improving and maintaining legacy systems, nor does he know how to do battle with Asshat. And as a symptom of all this, I'm starting to care less. Fuck it. If nobody is going to give me work or let me run with projects I think would be cool, I'll just do nothing. I'll do homework. I'll leave at 2:30. I'll learn things I want to learn, like R and Python.

I'm so frustrated. And I can't leave, because of my debt situation. Fuck this. I'm going for a run. Then I'll come back, but all I'm going to do is read my 100 pages of Markov Chains of class.

User Journal

Journal Journal: Adventures in Data Science

I knocked out a little bit of homework today. I've been doing my assignments in LaTeX (specifically ShareLaTeX), which is yet another skill I can kind of get for free while doing things I need to do. My handwriting is atrocious anyway, so I might as well. Plus, I'm in grad school. I should act like it. I'm going at a rate of about 2-3 hours per problem. It's rough. But I'm making progress.

In my spare time, I've been listening to Partially Derivative, a podcast about data science. I'm starting from episode 1, which is about a year old. They talked about this project, 1 CSV, 30 stories, where a dude takes one massive CSV (3.2 GB) and does some data science on it and tries to get 30 stories out of it. Well, a year later, it looks like he stopped a little bit short at 21, but good effort! I'm going through it and trying to see how he did it, starting with bootstrapping (the data munging part). Already, I'm seeing where the troubles arise.

We've gotta munge this text before we can do anything with it. And as we've all seen, human-entered data is never clean, particularly when dealing with plain text.

So I'm currently playing with that just trying to get to the point he starts at. It seems that Data Tools has changed a bit since his blog, so I'm currently trying to get it to work the way it is now.

I'm also working on a work-related DS project that involves a probability model. I'm not sure if I've got the math chops yet after just 3 weeks back in class, but I'm getting some ideas. I think that's helpful to get a real-world scenario to drive home the theory.

I also started reading Antifragile. Reeeeaaaalllyyyy liking the concept of this book. But anyway, gotta cut this one short. More later this week!

User Journal

Journal Journal: Dear Gournal 2

"Oh, you mean a journal?"
"Yeah, whatever. I guess I'm not all smart like you."
Really though, I'm kinda dumb. Yesterday I rolled my foot while out running. There was car trying to turn right. I was on his passenger side trying to cross the street and would have been directly in his path. And since I could see that he was looking left and had no situational awareness, was going to run me over, so I put the brakes on, only to have my foot find the edge of a sidewalk. I didn't go down, but my foot rotated 90 degrees and I heard a loud pop. That was that. Had to walk a mile home on a bum foot. I guess that doesn't make me dumb, I avoided getting hit by a car. But I wish I had just slowed my roll before getting close to the intersection. I may have also been distracted by some nice yoga pants-clad asses. Man I hope I can get back to running before summer ends.
I got through my first homework in my Probability Models class. That was a bitch. I spent probably around 48 hours of work on 20 problems. 2 of them went answerless. Sheesh. I need to get better at this though. I really want to be a Data Scientist. I mean, there are certainly times when I look at the math and think, Jesus, what am I doing?! I can make a fine living as a software engineer. But then when I hear about some of the awesome problems being tackled by Data Scientists, I fill up with excitement. That's shit I want to do!
The problem is, where do you start?
Obviously you need a solid understanding of math. So I'm starting in the right place by getting my master's and learning foundational mathematics. And the data wrangling/munging/ju jitsu. I think I'm pretty good there. I have a solid Linux background and am excellent at processing text. But then there are the technologies. They are a huge part of it and I know none of it. This is where I need to focus in my spare time. This is where I'm struggling with how to proceed. Do I do a brief overview of each technology, or do I deep dive one by one? Are there some I can skip?
This is my list of techologies to learn/understand:
* memcached
* MapReduce
* Apache Hadoop
* Apache CouchDB
* Google BigTable (proprietary)
* HBase
* MongoDB
* Amazon Dynamo (proprietary)
* Pig
* Apache Cassandra
* Voldemort
* Basho Riak
* Aerospike
* Google Dremel
* Google MegaStore
* Google BigQuery
* Google Tenzing
* Redis
* Apache Spark
* Apache Spark SQL
* Apache Giraph
* Spanner
* Apache Accumulo
* Impala
* Apache BigTop
Ugghhhh, that's not even scratching the surface. There's a whole section on Big Data and another on Databases at Apache. Sheesh. Where do I even start? I guess that's my initial struggle. Do I do an overview of each one, then deep dive, or do I just start deep diving from memcached on. Or do I just figure these out as I need them? Right now I'm playing with Pig. I exported a bunch of data and I'm trying to work with it. It's slow going due to the lack of documentation (at least from what I've found so far).
I just want so badly to get going on this stuff. To make it work for me and show my company that I'm a step ahead of everyone else. Well, anyway. I'll get there. Just gonna take some time and effort.

User Journal

Journal Journal: Systweak Scareware and Probability Models

One of my friends called me at work the other day and said, "So you know when you get a virus and you get this blue screen and it just says to call this number..." I was like, "No. No, that is never a thing."


Sometimes I forget that not all people know basic computer skills such as don't call numbers and get scammed out of hundreds of dollars. Thankfully, they hung up the phone and unplugged the computer just as things were getting hairy (they said the mouse was moving and the "tech support" guy was doing it).

So they brought it over and I took a look-see. It was surprisingly not terrible. It just had a bunch of shit from Systweak, a bullshit Scareware app that flags cookies as "threats." I removed McAffee since it had expired. Never fails, whenever I clean a friend's PC, it's got some old remnant antivirus that stopped working because they didn't buy the license. Just remove that shit, people. There's free shit that works just as well. So I installed Avast! and Malwarebytes, ran scans, and didn't come up with anything. Systweak was the only piece of shit contaminating the whole thing. The rest looked virtually untouched after a year of use. Good work! Let's keep it that way, mmmkay?

Side note: Windows 8 really is a piece of garbage. Fuck is that painful to navigate or do anything - literally anything.

I'm two weeks down on my Probability Models class. First 2 weeks of grad school in the books! Holy shit this is one of the hardest classes I've ever taken too. Way to ease back into things. The thing is, at the time I decided to go to grad school, this was about all I could take that I had the prereqs for. I don't remember shit from college. I mean, things are coming back, but I feel like Wolverine being fused to the adamantium skeleton. I'm going to be stronger after, but it's going to hurt like hell during.

I was kind of excited for this one because it seems pretty applicable to business, but so far, we've hit prob & stats I heavily, real analysis, calculus I & II, various theorems and inequalities not in the book, and the German Tank Problem. Uhhh, we're two weeks in. I'm still catching up on my prob & stats I and calculus. It's been anywhere between 5 and 10 years since I had some of these classes.

Don't get me wrong, I'm actually enjoying it. I spend just about all my spare time studying/doing homework, but I am not at all resentful. I think I'm just happy to have a higher calling. Something challenging. Something that I can think about, with all the bullshit of work, and remember that there is more to my future than the day-to-day. So I'm going to do this. I will learn Jensen's Inequality, Fubini's Theorem, as many random variable distributions as I can, fundamentals of prob & stats, calculus, Markov Chains, Queuing Theory... whatever else my professor throws at me. For I AM THE INFERNO. I AM LEGION. Little quote from Lamb of God's latest, which I've been jamming almost every day. Though as soon as it's released, I'll be pumping Mgla's new shit, Exercises In Futility. I mean, my God. No words. And such a great title for how I often view my personal and professional life.

Anyway, I need to sleep. Early-ish wakeup tomorrow. Got a clean laptop to deliver bright and early.

User Journal

Journal Journal: Back to School, Back to School, Prove to Daddy That I'm No Fool

Last night I came in to work to help with an oracle table space migration. The benefit being that we would get ~100 GB freed up. The DBA said it would be about a 20 minute outage while he rebuilt the indexes. Well, 20 minutes turned into 2 hours, because he neglected to think of the load on the production database. So I ended up having to shut down one of our heavier hitting apps and that fixed the problem. Still, I'm sleepy because instead of coming in, doing nothing, sending out an email, and then grabbing a cocktail, I came in, watched it not work, conferenced people in, made a decision that I hoped wouldn't fuck anything up, and left with enough time to slam a shot and a beer. Ugh, nothing ever runs smoothly.

Except my deploys when no one else is involved. That almost always goes smoothly. It's relying on other people that makes things difficult.

Today I start grad school. I've been getting amped about this Data Science thing. I remember waaayyyy back to when I was a C Programming student, and I applied to teach the C++ companion class. During my interview, the instructor asked me, "Do you prefer to work with People, Programs, or Data?" Assuring me there was no right or wrong answer (though in hindsight, "People" was probably the better choice of the three), I said Data. I was being honest, and I kind of said it without thinking. Coming back around now and thinking about it, yes. Yes, I do prefer to work with data. I want to munge it, wrangle it, ju jitsu it, analyze it, model it, and present it. I haven't been this excited about something since I started my Bayesian text classification project, which, come to think of it, was really a data science project. My calling has been sitting in front of me this whole time.

But I'm glad I went this route. I've become a good developer, which I think is paramount to being a good data scientist. I suspect that a lot of data scientists are not proficient in software development, primarily because it isn't the central focus of the field. But I think that will bring a lot to the table.

Looking at the data science curriculum, I'm a little concerned that it doesn't really bring in any of the major data science tools - R, MapReduce/Hadoop, NoSQL, etc. It seems like it's going to be a purely mathematical ride and I'll be left to my own devices for the tools. Which is... I mean, it's fine, the math is the hard part anyway, but I'd like a little instruction on some of these tools. I've been taking Coursera courses to try and get an high level understanding of all these tools. I just hope I get a little practical knowledge thrown in with all the theoreticals and foundation.

R is looking like a really cool fucking language. So far from the little bit I've played with it, it looks like it will cook you breakfast while giving you a beej. I absolutely love read.csv and read.table. Makes working with data so much easier than, say, Java or ColdFusion.

Well, anyway. I'm rambling at this point. But I am excited. Let the games begin!

User Journal

Journal Journal: Every Time I EDI

As I mentioned before, I'm working on a RMM Level 3 RESTful service for our partners to use for B2B transactions. It is made up of resources and actions on those resources. But that's for the future. Right now, I'm stuck dealing with the less than stellar "Company R" as I will call them. They're primarily an EDI shop, like most of the partners I deal with. But they've been instructed to use web services. To them, that means "EDI over the Web."

So as a compromise, I've build a couple of additional services that handle the two specific types of EDIs that we need. I take the EDI in its raw form as the body of an HTTP POST request. Then I parse it using the wonderful GPL'd EDIReader.

Unfortunately, there's no Maven version for this, and it's hosted on the archaic Sourceforge. Ahh, old and decrepit, just like EDI. I have no idea if BerryWorks is maintaining this, or what, so I pulled the code into a new shared component that I'm calling edi common. EDIReader uses SAX to parse an EDI into XML format. So once I do that, I use JiBX (hello darkness, my old friend) to unmarshal the XML EDI into a Java object. I call this my BaseEDI. Since I know what kind of EDI I have, I can then get more specific, and write helper methods to get exactly what I need from a particular element on a particular segment.

All of this is very ugly, but I'd like to think that I've done it in the best way I could think of. And it works phenomenally. I just put it into test this week and I'm writing a testing tool to help with the creation of EDIs.

I have no idea when this thing will go live, but it'll be ready for whenever Company R decides to get with the program.

They have not been so good at even traditional EDI, so migrating to the web is only going to have the added benefit of getting them real-time error messages. I have surrounded my code in so many error catches with specific error messages to really hone in on where problems may lie. It's not even a bit fun to read through, but I know if they start failing, I'll be able to point to the exact reason and tell them immediately. And as in the past, it will not likely be my code that causes the problems, but theirs. I had so much fun during our first Go Live looking at errors and saying, "it's on your side." heh. There's a certain satisfying smugness in that.

Anyway, this week we also learned of impending layoffs at the company. Fuck. No good will come of this. I hope I've done enough to cement my necessity to the company. But slightly more likely is seeing my manager go, or anyone else I've enjoyed working with. That has the ability to fuck all kinds of shit up. If I end up working for Ass Hat, I'm going to lose my shit. I guess I will keep my fingers crossed until then. But I will continue to drive on and keep churning out great products that people love in the meantime.

User Journal

Journal Journal: I Push the Money Over Here, Then I Push it Over There

In the last 5 years, I've become something of an expert at managing my personal finances. Not that my finances are in great shape - it is what it is - but at least I'm good at working with what I have.

I've done this with a combination of technology and an incessant desire to know everything about my past, present, and future, especially financially.

I use Simple for my banking, and I love it. It gives me a clear picture of what is happening with my money in real time. The analytics are amazing. Of course, analytics alone can't work magic. You still have to analyze the analytics. For instance, I could get an average of what I spend on food monthly from the time I started using Simple and you'd think that'd be a pretty good estimate, but alas, it is not. The picture has changed from 2013, because I gained a live-in girlfriend in 2014. And now this year, that girlfriend has started Anesthesia school, so I am paying for all food for both of us. So the picture has changed. I think this kind of analysis is good for my Data Science future, but also is good for keeping up to date with myself and planning my future.

To do that, I have Google Docs spreadsheet that spans from 2001 to 2020. I am able to capture most of the financial picture of my past using old bank statements, standard military pay charts, and memory. This spreadsheet has become my budget, and a predictive model for the future. Using past bonus and raise percentages, I can extrapolate out to the future to determine (if all goes well) what I should be making in 1 - 5 years. Furthermore, I can determine how to most effectively manage my debt. My loans are all calculated by APR, so it even takes that into account.

Unfortunately, sometimes cold hard analytics fall out of the way in favor of anxiety and stress, which is often the result of some of this analysis. When you move the dials all you can, and there is still a negative sign in the leftover balance of the month, you realize the best option, financially, is not the best option sociologically. So while the Dave Ramesy cult followers of the world will tell me that halting payments into my 401(k) and even worse, borrowing from my 401(k) to get through some rough times is a terrible idea, I would argue that they are fucking idiots because not all situations are that goddamn black and white.

Obviously I don't prefer this plan. But I've spent the last month having absolutely no fun, going crazy in my apartment, ducking friends because I can't afford to hang out, while my girlfriend does the same - and this is causing stress, and stress causes unhappiness. And friends, you cannot put a price on happiness. We're not even talking about happiness here, it's contentment. Just getting by and not hating life. It's a big fucking deal. I cannot do 1-2 years of this.

So I pulled the trigger, just now, on taking out half of my 401(k). I had ruled out this option in the black and white scenario. But a crying girlfriend can change everything. We will not live in poverty, or be ashamed of not being able to see friends because we can't afford it. It's ridiculous. I realize this is First World Problems, but you know what it isn't? 1% problems. This is 99% problems. I'm not a big Occupy Wallstreet guy, and I don't think we should just go take people's money and redistribute it all Commie style, but you have to admit, the game is stacked pretty strongly in favor of the fucking wealthy. It's a goddamn logarithmic curve. Getting over the hump is damn near impossible, but if you can, the rest is easy. Money begets money. And you know what begets poverty? Poverty. It is WAY more expensive in America to be poor than it is to be rich. And that is the fucking travesty of our time.

If things continue on track for me, I will be over the hump of the logarithmic curve by mid-2016. This hurts to think about, because I had every intention of being out of debt by this month a year ago. Plans change. Situations change. Anesthesia school is expensive. Grad school is expensive. Food is expensive. There's nothing I can do about it but continue to plug along. But I can't live like a pauper. I've worked too hard and for too long to go back to college kid life. Plus, I'm in my 30s, I hate college kids.

I can't change the rules of the system we live in, so the only thing I can do to get over the hump is to beat the system at its game. Which is why I'm going back to school, and why I'm supporting my girlfriend in what could be a very lucrative career. As disappointed as I am to see success put off for a few more years, I know it will come. Patience and resolve will get us there. Until then, we'll just keep doing what we're doing.

Everything will be ok.

User Journal

Journal Journal: IT Heroes and Big Data

Many moons ago (circa 2006), when I was a young lass studying Computer Science, I had a teacher that really stood out. He swore freely in class, wasn't afraid to call a kid stupid for asking stupid questions or not being able to read code, and regaled us with probably one too many fantastic stories about how he played football for the Chiefs. And while he could be an asshole at times, I probably learned more from him than just about any other throughout my college years.

He didn't just teach C/C++, but lots of other practical tidbits along the way, such as Bash command completion, exiting a shell with CTRL+D and why that works (signals end of input), using Splint, and a myriad of other little things that where maybe outside the scope of the class, but added a lot to my IT tool belt. This made him one of my first IT heroes.

Of course there are others that I have never met and are well known throughout the community - Bruce Schneier, Theo DeRadt, Ken Thompson, Donald Knuth, etc. Those guys have all earned their place at the top of my IT heroes list.

So how does one become an IT hero? It seems there are two paths. One can teach, or one can code their way to fame. I'm not sure that I'll ever be anyone's IT hero, unless I get a cool intern someday who is eager to sponge up knowledge. So far, most of the interns I've been around don't remind me of me (i.e. they didn't come in with a breadth of knowledge, and they don't seem super interested in tackling big projects to expand their abilities). Maybe someday...

On a completely different note, I am loving this Data Science class I'm taking on Coursera:

And I'm also a little ashamed that it's taken me this long to learn about MapReduce and NoSQL. Really ashamed, actually. There are so many problems I could have solved with these tools had I known about them years ago. Of course, we at the big corporation have only recently begun to embrace Hadoop, while NoSQL has not yet entered the enterprise at any scale.

And with my current project, I'm wondering if there's a place for it. I'm writing a Hypermedia API and I want to track requests, responses, failures, etc for analytics. In our existing internal RESTful services, we're storing entire JSON objects in Oracle CLOBs. This has of course started to bog down Oracle and chew away at disk space. And it's a bitch to even try munging for analytics. In fact, SAS is doing a PoC on that data, so we had to parse it for them and put it into a table. So that whole thing sucks.

So I'm like, 95% sure I don't want to go that route, especially since that database is already having growing pains. But I want to capture that data. But (yet again, "but) I am just learning about NoSQL and Hadoop, so I don't know if I should try and incorporate one of those systems into my design, since I know my first implementation is not going to be great. There is the option of writing to flat files and saving those off somewhere for later parsing, but I know that once I get this thing set up, I'm probably not going to get a second chance to implement this. A new project will come along and this will be forgotten about until something breaks or someone asks me for stats.

I'm currently on the NoSQL chapter in the Data Science class, which is basically a 50k ft. view, but it's more than I've learned about NoSQL up to this point. Again, I'm a little ashamed that I haven't dedicated some time to it up until now. Better late than never though.

We shall see.

User Journal

Journal Journal: Less Hate for HATEOAS

I'm starting to finally warm up to this whole Hypermedia thing. It's really not so terribly difficult to understand.

I actually figured it out on my way to the Slayer / King Diamond show. It was a bit of a walk for me, so I use that time to think through my projects without being at my desk. Helps get me outside the problem and see the forest for the trees.

It finally occurred to me that we're using controllers to provide methods. Those methods either return a resource (a mapping of an entity), or perform some kind of action on a resource. The action performed is declared by the client in the HTTP Method. GET returns a resource. POST and PUT do some kind of modification to a resource (the difference being POST is not idempotent, a word that I have just this week learned).

So on my walk, I realized that everything in a GET is just a resource. Why not just name them all "resource"? The client should be able to figure out what kind of resource it is based on the URL. I think the key to reaching REST glory is understanding that HTTP IS THE FRAMEWORK. Every REST service I've written up to this point has been like, "/service/something/do" or "/stuff/morestuff/get". Why?! There should be no verb in the URI. That comes for free with HTTP Methods! And all these verbose responses with error codes in XML and shit? C'mon, that is given to us in HTTP Response Codes! And I think once that realization hit me, everything came together (in my brain, still lots of coding to do).

As a previous comment asked how I am serializing XML, well, it's with JAXB. Seems to go hand in hand with Jersey, so I'm not going to rock the boat. I actually briefly tried Spring HATEOAS, and it didn't play well with Jersey. Couldn't find the servlet context, probably because Jersey had it. I dunno, didn't dig too deep on that. It's such a simple concept here, presenting links to allow for a responsive API and client, it wasn't hard to roll my own.

So all I did was create a HypermediaLink object that just holds basic string stuff like rel, and href. Then I created a BaseResource that holds at least a selfRel and a list of links. My resources extend this, but here's the dick kicker - the @XmlRootElement(name="resource") has to go in the inheriting class. No way around that with JAXB, unfortunately. So that's not super clean, but at this point, we're splitting hairs.

So armed with that plan, I drew up my API state diagram. It's still a work in progress, but all the sudden it feels SO CLEAR what needs to be done. So now I just need to do it. Hard part is over though. I'm actually very happy with my work, thusfar. Looking forward to banging this out and chalking up another huge win.

User Journal

Journal Journal: Hating HATEOAS and a New Whiteboard 6

I'm designing a REST API kind of from scratch. Actually, I'm taking an existing API in production, and putting kind of a "Version 2.0" on it, leaving the original in place, but making the v2.0, well, better.

So I've decided to read up on REST and instead of just attacking it like I have every other RESTful project I've done, actually approach it with an architectural mindset with some theory to back up the practice. So I read about the Richardson Maturity Model, about how most of my services up to this point have been Level 1 services, which is fine, since I was the only consumer of them. But now that I'm going to be exposing this to the world (at least a subset of the world), I want it to be done right. At least, as "right" as I can get it given all the conflicting information on REST.

The biggest problem so far is, I'm building this with Content-Type=application/xml, not Content-Type=application/json. Why? Because I don't think our consumers are going to be big JSON users. To that end, I don't think they're going to be big HATEOAS consumers either. However, I want to do it this way anyway, so no one can say we weren't forward thinking. So back to the XML problem - there are like, ZERO examples of an XML Hypermedia API. Everything is in JSON. Even the RFC for HAL is for JSON. Why not include XML in that? The two go hand in hand. It's just a different dialect for the same data.

After exploring two frameworks for HATEOAS (spring-hateoas and Jayway Forest), I found that there's nothing that will support XML. So I'm rolling my own. I guess it's pretty simple, we're just representing links, right? But it's a little bit trickier because link displaying is dynamic. So I need a way of conditionally displaying links. I'm leaning toward annotations. I got this idea from Jayway Forest. That seems pretty basic.

And it's looking like the closest thing to a standard is HAL+XML, which isn't even an RFC yet, but a guy on Google Groups who just modified the RFC for HAL+JSON and posted it to Github. Man, this is looking ugly. So, in light of this, I'm thinking there really is no standard, and I'm just a well off writing my own as trying to follow someone else. So that's what I'm doing.

Previously we were returning XML that looked like this:

        <status-code />
        <status-message />

There are a bunch of problems with this. First of all, the root element changes for each Controller. That makes it very hard to write a client.
In fact, calling it any kind of response is incorrect. That is a Java developer habit. We write everything with services, so everything is a request/response. But here, the response is the entire package - HTTP Headers, Status Code, and Payload.

Secondly, why is there a response header in the response body? We get that for free with HTTP! Get rid of that thing and we're already starting to creep into RMM Level 2!

What we're representing in XML is the Entity, or Resource. A Resource is just a representation of an object's state. Like a lightbulb:

<resource rel="self" href="/room/kitchen/lightbulb">
  <link rel="switch" href="/room/kitchen/lightbulb/flip" method="PUT" />
  <link rel="change" href="room/kitchen/lightbulb/change" method="POST" />

One thing that isn't in the "spec" if you want to call it that is the method there. That at least tells the client what I'm expecting for the verb.

I'm really torn on all of this, as I'm sure many other developers trying to implement this kind of thing are. Nothing is concrete and there's pitfalls everywhere. I know it's been like this for a long time, but it just seems like the more it gets refined, the less anyone seems to know what to do with it.

Oh, and my whiteboard got replaced today with a smaller one. The super annoying thing? I had my whole REST API written up on it for reference. Motherfuckers.

User Journal

Journal Journal: Weird Ubuntu Wireless Shenannigans 2

I got a call from the gf the other day at work. She was at home doing homework on her Ubuntu laptop and said the Internet wasn't working. I believe she said she couldn't connect to the Wireless, actually. So I had her turn off the router and turn it back on. When that didn't work, I said I'd be home in a bit, because trying to walk her through troubleshooting is a nightmare. So first thing's first, I tried pinging Google from my desktop, which also connects wirelessly. I'd get about 39 pings off and then it'd freeze for about a minute. Then I'd get some more pings off and it'd freeze again. I powered off the modem and powered it back on. Same thing.

So I didn't have time to dig into it anymore, so I told her to just come to my work and she could use the office WiFi.

When I got home, I tried again, and everything was hunkey dorey until I turned my laptop on. Then I saw the problem again. I set both the laptop and desktop up to ping Google and I'd get the same thing - on again off again connectivity, but they would alternate. Ah ha! I had a theory. So then I pointed them both at and got the same result, proving it wasn't the Internet.

The theory was gaining traction.

I pointed my desktop to the router's admin console and checked my devices. I saw there were 3 connected (my phone was on too), but only two were identified with an IP address, and the other was just all dashes (-------- ----------). So it saw a device, but couldn't assign it an IP. I checked all the settings to make sure my DHCP wasn't set to only allow 2 devices (something I have done in the past), or that I didn't have any static IPs set. Nothing.

I grew more frustrated.

Then I disabled networking on the laptop. When I turned it back on, problem solved.

So this must be a weird Ubuntu bug, or something related to the driver for that card. I dunno. But I'm glad it's fixed. That would have driven me nuts for a long time to come.

Slashdot Top Deals

Quantity is no substitute for quality, but its the only one we've got.