Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
User Journal

Journal Journal: MD5 of Song Data: Finding Duplicate MP3's 1

Update: I started this project but still can't strip the ID3 tags temporarily. I'll put the details and code snippets in a post.

Update 2: I more or less gave up on stripping the MD5 tags for now since it will take too much work. However, all ID3v1 tags are at the end of the file. ID3v2 tags can be and usually are at the beginning of the file, but ID3v2 tags are far less used in my collection, so I used the following command to get the MD5 sum of the first 500KiB of the file. It will miss identical songs that have had ID3v2 tags at the beggining of the file added or altered, but I was able to delete another 1.2 gigs of duplicate mp3s with this test:

head -c 512k <mp3 file> | md5sum -b

md5sum in this command returns a filename of "-", but I used some shell trickery to figure out which sum went with which file.

***************

Foreward: Really my only question here is "how do I calculate the MD5 sum of MP3 song data only--that is, not including ID3v1 or ID3v2 tags?" The rest is just chatting about what I'm doing.

I'm a network admin who thinks he knows enough about programming to pull this off. (I had several programming courses in college and occaionally play around with one language or another but never had a developer job or a serious project.)

I have disorganized, reorganized and relabled MP3 files. I want to find duplicates. Here's my idea to accomplish this:

I'm going to copy information from each MP3 into a database, either directly or through an intermediate delimited text file. The major info will be the path & file name, MD5 sum of the file, and the MD5 sum of the data only (minus the metadata/ID3 tags). I'll probably also try to get the MD5 sum of the first X seconds or X Kbytes of song data to detect duplicate beginings (in case of a truncated duplicate). Secondary info will include all the other stuff like file size, song length and ID3 tag data; this info will just be for me to locate duplicates that aren't byte-exact and to help me decide which of the duplicates to delete.

I could probably do this with a shell script and a couple of utility programs, but it may be simpler to grab a couple of modules from CPAN and use Perl. I don't anticipate having trouble with the metadata or MD5 sums of the entire file, but I don't yet see an easy way to calculate the MD5 sum of only the song portion of the MP3. I'm browsing through CPAN, and there are tons of modules which read and edit ID3 tags but nothing quite like what I have in mind. I did find one that strips the ID3 tags, but it seems to alter the file directly, and I only want to have the tags stripped just long enough to get the MD5 sum.

Sure I could read the data structure of the MP3 file and parse it myself, but I intuit that somebody made a perl module or command line utility that can nondestructively present me the tag-stripped MP3 so I can MD5 it and leave the original file unchanged. Anybody know of such a module or utility?

My app won't be pretty or user friendly. It will probably take a path as input and output either a delimited file or update a mysql or pgsql database directly. Then I will query the database to find what I need. I'm not only finding duplicates; I'll also be looking for files that are unique per storage device, because I have most of my MP3s on my main PC and my MP3 player (w/20G HDD), but there are some MP3s on one device but not on the other and vice versa, not to mention some on my linux box and some on my work PC. I can import from all sources into different tables and play with queries and eventually consolidate and catalog my mp3 collection.

I know there are some various apps that try to do some of these things, but I like working at the command line and having my hands on the raw data; every time I try to deal with an MP3 manager application I hate it.

EDIT: Yeah, I could just delete all my MP3's and re-rip them, but what fun would that be? Mucking about with Perl and SQL sounds so much more fun.

User Journal

Journal Journal: Nevermind (was: Where did that article go?) 2

Found it! Rekall. Slashdot article here, sites here and here. Also I now remember I wanted to try out OOo as a db client, too.

*******************
Sometime recently, Slashdot ran an article about a generic database front end that went from proprietary to open source. Kinda like Access but without a database engine--just the user interface. I meant to come back to it later and try it out, but I can't find the article or remember the name of the product! I've searched Slashdot, Freshmeat, Google, Google Groups and Google Directory with all sorts of combinations of keywords like database, front end, mysql, postgresql, report, designer, query but had no luck in finding the article or the software.

I think there was an R in the name of the software.

Does anybody remember that software or article? I think it was in the past couple of weeks.

Thanks!

P.S. To avoid smartass responses like the one I would leave ("Yes, I remember it" with no more info), I also request that you name the software and/or link to the article and/or the software site. :-) Thanks!

User Journal

Journal Journal: I Still Exist 2

Hi. I'm still here. Nobody's pregnant, engaged or dating underage girls. Nothing interesting going on. Well, work is totally shaken up, but I don't feel like whining about that anymore.

Oh, since my last JE was about T I'll update: I called her a couple of days after the last JE and the conversation went downhill so far that I never called her back.

User Journal

Journal Journal: Return To the Blues Bar With T 4

As I said in last entry I called and left a message Weds 9-10. I didn't get a call back that night, so I wrote her off. I had family visiting for the weekend so I didn't go out.

Sunday morning some girl called and left a message, but it was hard to make out and she sounded timid. I thought it was T but wasn't sure. I still had family in town and didn't feel like calling her back.

I kept putting off calling, and I figured if I hadn't called her by now that I'm probably not as interested as I thought, and then Tuesday--almost 2 weeks after my first call and over a week since the possible return message--I figured what the hell, I'll just call and see where it goes. (That was her that left the message Sunday.)

We met again Thursday night at the blues bar; she brought her friend C along again. Actually C drove her. We hung out, danced and had fun. C wanted to leave early and T didn't seem ready to go. I thought I was being a bit too bold when I offered to drive T home, but she jumped on the suggestion immediately and told C "bye".

We were talking and having a good time, and when we got back to my car I asked if she wanted to hang out anywhere before going home. T seemed noncommital, and I couldn't think of anyplace decent to hang out so I suggested (too boldy, I thought) my place. She was fine with that.

We went back to my place, had a snack, hung out, listened to music and cuddled a lot. (She started it.) She stayed all night, and we got a bit friendly but probably not as friendly as you might imagine. Sorry, no details, but T does seem a little conflicted. Then again I'm starting to think she thinks I might be a player (LOL; the thought cracks me up), so maybe she's being cautious. This was only our second meeting, after all.

User Journal

Journal Journal: Poll: When to Call? 2

I planned on calling T Monday night and seeing about scheduling something Wednesday night because I have family visiting this weekend. But I worked late and got home too late to call.

Tuesday at work I was given the advice to not call her that soon, but to wait a few days. I had met T Saturday, and the advice--from a woman--was to wait to call until Wednesday or Thursday.

Between the advice, some dental work and my family not calling to confirm plans I decided not to call T Tuesday night. (I wasn't sure I'd be able to eat a nice dinner by Weds night; turns out it would've been okay.)

Tonight, Wednesday night, I called but got voicemail, and I left a message and my number.

Anyway, the "when to call" advice got me wondering. I'm not particularly worried about it, but I figure the next day is probably too soon, and two weeks is way too long. What do you think?

After meeting a girl in a bar and getting her phone number, call her back:

A. 30 minutes after parting for a booty call.
B. The next morning.
C. The next day.
D. 2-3 days
E. 4-5 days
F. 6-8 days
G. 9-14 days
G-and-a-half. 4 to 6 weeks
G-and-three-quarters. When really drunk, desperate and lonely.
H. Sucka! Gonna show this # to friends and post it on the trophy wall!
I. Next morning, push her out of bed and tell her to make breakfast.
J. Doesn't matter, 'cause it's a fake number, anyway.

User Journal

Journal Journal: Geek at the Blues Bar 3

As seen in my most recent JE I had no luck last night in meeting women or even striking up a conversation. When leaving the parking garage, though, I passed a Blues Bar that boasted live entertainment. I recognized the name as someone had mentioned it to me before. I made a note to come back to it sometime.

Tonight (Saturday night) I almost didn't go out. I was discouraged by Friday night's utter lack of interest and the money I spent failing so. I even played some Quake II for a couple of hours, and the sun went down. But I decided I really should go out, but this time I would do something that I enjoy more than just trying to meet women. So I looked up the blues bar's schedule and saw that their bands started at 9:30 tonight. Cool! I like late starts.

So I played the computer, ate and showered. I decided just to wear a t-shirt, old jeans and tennis shoes tonight because I wasn't really trying. As I got in the car I noticed the shirt had a faint smell of mildew; oh well, I wasn't planning on getting close to anyone, and maybe the smoke at the bar would mask the smell. I arrived at the bar right as the band started. Oops, no seating room. So I stood, but the band was great.

I was standing where everyone had to walk by to enter or leave the stage room, so I got to watch a lot of nice women go by. Nobody seemed interested, though, but I was there for the music tonight.

At the start of the second set two black ladies stood near me to watch. They were having fun and we talked a bit. As the band played on we danced in place a bit and talked some more. I bought them a round of beers. At first I thought the one with the cuter face (we'll call her C, TechnoLust-style) might be interested because she was closer to me and started brushing against my arm while dancing. It quickly came out that she was married, but unhappily and seemed to be acting single. That turned me off, because I don't want to mess with a married woman; if you don't want to be married, get divorced but don't stay married and play around.

The other girl--the one with the nicer body, who we'll call T--then seemed much more interested in me. She told me I was cute. She and I started talking and she talked a lot. Through the night it became more apparent that she liked me. I was a bit apprehensive because I haven't been out with a black woman before, and her face--although not ugly--wasn't particularly attractive to me, but she had a good personality and a nice body and very smooth skin.

Maybe they were both into me, because at the end of the night T got more physical with me and then C suddenly wanted to leave. T gave me her cell phone number, and we have tentative plans to go to a jazz restaurant.

=====

While I was hanging with T and C and we were going outside for a smoke/cool-down break (T and I don't smoke, but it was hot inside) a very pretty woman--better face than C, better body than T, my skin color, and a funny personality--made an amusing quip about the way I looked at a statue as I passed. I don't know if she was showing interest or just having fun, but I don't think I could've started a conversation with her without pissing off T and C. It was tempting to try, anyway. D'oh! I just realized I should've tried to find her after T and C left. At the time I was tired and decided to go home myself, and my mind was racing with analyzing the evening and debating on wheter I really want to take T out or not. (I think I will.)

User Journal

Journal Journal: MyHair, the Bar Hopping Geek - or - Sex and the Single Geek

I am almost the stereotypical Slashdot dude. I'm very overweight, spend too much time on computers and haven't (before last weekend) dated much or got laid in years. I don't live with mom, though, and in fact I live alone and am financially independent. Sorry.

Last weekend on a spur-of-the moment roadtrip mini-vacation I got laid for the first time in...well, a long time (at least two years). That helped clear out a lot of the anxiety I had built up about women, increased my confidence/self-image and greatly increased my desire to start dating again. The girl I was with lives about 6 hours away, and it was more of a one-night thing than a long-distance relationship type thing.

So I went bar hopping. I've never done this before, even when I was actively dating. I went to a TGI Fridays at 4pm on Friday and stayed until 6pm. This particular place had been recommended to me by a couple of guys at work, but only couples and much older (late 40's to 60's, and I'm 33) people came in. After eating an appetizer sampler I played the NTN trivia game with some of the other guys until 6, then decided it wasn't going to get better and headed downtown.

I walked around downtown a bit before going into a huge bar with two restaurants, 4 bar areas, a video game area, a dance club and cosmic bowling (bowling plus black lights, music and projection TVs showing crazy stuff). Nothing happening in the restaurant/bar area, so I went through the video game area. It was early enough that kids were still allowed, so lots of families here. Nothing happening in the bowling area, and the dance club was in the middle of a private party, so I went back and played some video games.

As it got later, all the kids disappeared (I think they kick all the kids out after 9pm) and more adults showed up. Most of the women had dates, and I wasn't getting any eye contact or any other signals. The only group of women that didn't have guys around looked gay and didn't look at me once, even for a second. Plus they were french kissing each other.

I went back to the bowling area; nothing still. Went to the dance club, but nobody was dancing, and there were 2 girls and about 100 guys. (The private party had ended.)

Back to the video game area. I got some eye contact on the way in from one of three girls who just came in. Cool. I walk scout the area a little more to give them time to settle in. The go to the bar and order drinks. I go to the bar and order a drink, but they're not looking my way now. One is a really gorgeous blond, but the other two are nice, too, and one of them had looked at me. I don't think I've had enough signals to approach with success, so I'm watching, waiting an looking around, but three guys come up and join the girls, and the girls are obviously very into them; so much that I think they might've arranged to meet there.

It's getting late and I'm tired, so I leave and head home, stopping at the Wal-Mart to get some stuff for the apartment I've been meaning to get.

Well, I suppose it's a halfway decent first step to becoming more social even though the only conversations I struck up were with the bartenders, the NTN trivia players and a (male) coworker I ran into at Wal-Mart.

(No, the lesbians weren't really french kissing each other. Everything else is true.)

Caldera

Journal Journal: ESR Says SCO Massively DDoS'ed, Please Stop

I'm submitting this story shortly. If it's posted you can leave me feedback here. If not, you can discuss it here.
=====
Much speculation is found on Usenet and Slashdot about why the SCO and Caldera web sites have been down Saturday and Sunday, but Eric S. Raymond says they are being hit by a "massive denial-of-service attack" and asks the attackers to "Please stand down *now*."

User Journal

Journal Journal: Friends and Foes, Updated 1

Ah hell. I had it all figured out. I've been adding more friends lately because it's easier than browsing through friends-of-friends. So I started moving my 4 'tech interest' friends to my foe list--not to be real foes, but to bookmark them separately from 'social interest' friends. Then I realized my little stunt does actually affect others. I have fans now (hi fans!), so whoever I add to my foe list will show up as foe-of-a-friend when my fans are reading their posts. So I felt guilty and put my not-foes back as friends. So much for my clever ideas. And I'll probably recklessly start adding friends now. I guess I'm officially out of control. :-)

User Journal

Journal Journal: My Take on Friends, Fans, Freaks and Foes 1

After reading some comments in bethanie's and Elwood P Dowd's journal entries about Slashdot friends I get the impression that some people feel it's a give/take thing, and friends/fans should be mutual.

I don't think that way, so don't be offended if you made me a friend I don't return the action.

I'm using the Friend function more like a bookmark. I've added a friend when I think they'll periodically have posts or JE's that interest me so I can quickly locate said posts/JE's later. I'm also enjoying the friend-of-a-friend feature to locate other JE's and posts since my chosen friends seem to have chosen interesting friends, too. And I also check my fans from time to time, too, although many are on the friend-of-friend list, anyway.

I haven't done it yet, but I've considered using the Foe list as a second interesting bookmarks list. I might want to separate social interests from technical interests, so I could put Bruce Perens, Sanity, fv, and MikeRepass on one list for tech interests and TechnoLust and Surak on the other list for social interests. I just realized there is a "Foes of Friends" but no "Friends of Foes" function, so I guess I'd put the tech interests on my Foe list and the social interests on my friend lists. On the other hand, since I have only two social friends I could make them Foes and just click to their friends lists individually.

Well I started babbling, but all I'm saying is don't take it personally if you're not on my friend list or even if you end up on my foe list!

(Yeah, yeah, there's "Freaks" in the title but none in the message...so sue me.)

User Journal

Journal Journal: Tiny OSes & Some Misc. Words 1

This JE is really just a bookmark for me to refer to later. The comments in this story mentioned a few small OSes for old PCs that I may want to toy with sooner or later.

http://developers.slashdot.org/article.pl?sid=03/08/06/1249252&mode=thread&tid=185&tid=190

Oh, and just to throw something else out there, I think the fact that this post got modded to +5 Funny is both funny and sad. (Keep hitting parent link and you'll see it's a long lineage of +5 Funny posts.) I appreciate it anyway, though.

Hi to the 'journal friends' gang if they're reading. Several months ago I added TechnoLust as a friend because I found some of his JEs interesting. He seems to be part of an unofficial/undeclared group of Slashdot friends who post JE's and comments to each otehr JE's. Just yesterday I added Surak as a friend for his amusing and embelished weekly recap JEs of other JEs/comments.

Hi to my fans! But you people should really seek professional help. :-)

I have a freak now! I'm not sure where I picked him up. Maybe it was that post that got modded -1 Flamebat (Profane flamebait title warning.). And I thought it was hilariously funny. :-/ Then again I thought this was funny, so apparently I dance to the beat of a different drummer.

Music

Journal Journal: RIAA and Filesharers Both Wrong

I just found and submitted this story. In case it's not posted, here it is. In case it is, you can leave me feedback here about it.
---------------
The Motley Fool's Rick Aristotle Munarriz takes a business perspective of the RIAA and the filesharers, says "both sides are wrong" and presents a possible future for fans, artists and labels where the labels get a cut of the artists' merchandise and concerts, and CDs become a promotional item rather than the main "bread and butter". Filesharing would also be promotional.

User Journal

Journal Journal: Comment Moderation

I find Slashdot amusing, and I find my quirks amusing. I went for a few weeks hardly posting anything, then the past couple of days I've been posting like crazy.

In most stories I usually have nothing particularly insightful or interesting to add that hasn't been posted already, so the posts I do make tend to be either funny (well, intended as funny) or me babbling about something in someone else's post that piqued my interest.

I'm not a karma whore or troll, but I do enjoy positive moderation. I'm usually surprised by what gets positive moderation and what doesn't. If I think a post is going to be modded funny or insightful or informative, it usually isn't. Then yesterday I ramble on and on about a previous job in a post that I thought had only a small chance of being slightly interesting--to the parent poster only--and it gets modded to +5!

The difference seems to be that in the high-expectations/low-results posts I'm trying to get points or at least trying to get my quip into the mix, while in the surprisingly well-modded posts I'm speaking off the cuff about personal experiences and any insights I had about them.

Perhaps it's an area-of-expertise thing: I'm rarely the most savvy person about the posted story, but I know about my experiences. Then again maybe it means I suck as a comedian. :-)

Slashdot Top Deals

One possible reason that things aren't going according to plan is that there never was a plan in the first place.

Working...