Today I was presented with a problem. A colleague of mine did a MySQL dump on a database. The resulting MySQL dump file was 40GB. Problem is, he forgot to tell mysqldump to leave out the drop/create table statements. Here’s the rub:

What is the most efficient way to remove the first 43 lines of text from a 40GB text file?

“Surely there must be a simple *nix utility to do this.” we thought. My first guess was simple:

me: “Just use sed and tell it to delete the first 43 lines. Simple.”

colleague 1: “But won’t the regex engine be very inefficient for that?”

colleague 2: “Meh, I think sed is the right tool for the job.”

Now at this point we have a square off. Colleague 2 feels that the inclusion of the regex engine is simply too much overhead and is like bringing a bazooka to a knife fight. So they start to look around and I start doing some tests with a 1.5GB MySQL dump from Killoggs. Immediately, I see that it’s trying to create a temp file. If it’s creating a temp file, then that means it’s going to either create the temp file and rename it or create the temp file and copy it. Neither of those are especially good solutions because it will result in (at least) a lot of CPU utilization and even after the basic operation is done, there is still more to do in the way of a copy, move, etc.

So the next thought is doing the inverse. Rather than trying to delete the data, maybe we simply don’t want to output it. This logic seems to bear more fruit and the result is the following two potential solutions:

• a) sed -n ‘43,$p’ filename | mysql
• b) tail -n +43 filename | mysql

Now, the real question? Which is faster? To test this i created a 200MB dummy text file comprised of log data. I then took this and ran the following commands repeatedly. The result? Both sed and tail were about the same as far as both real and system time. They both fluctuated and both came out roughly neck and neck. The real key was user time. Sed was always an order of magnitude slower as far as user time. In the end, this would lead me to concede my initial suggestion and go with tail.

Got thoughts on this? email me at brian …at… logjamming.com!

[bharrington@berstuk tmp]$ time `tail -n +41 test.log > /tmp/file1.test `

real 0m6.479s
user 0m0.077s
sys 0m1.145s
[bharrington@berstuk tmp]$ time `tail -n +41 test.log > /tmp/file1.test `

real 0m4.454s
user 0m0.061s
sys 0m1.091s
[bharrington@berstuk tmp]$ time `tail -n +41 test.log > /tmp/file1.test `

real 0m4.498s
user 0m0.049s
sys 0m1.058s
[bharrington@berstuk tmp]$ time `sed -n ‘41,$p’ test.log > /tmp/file2.test `

real 0m5.025s
user 0m1.434s
sys 0m1.063s
[bharrington@berstuk tmp]$ time `sed -n ‘41,$p’ test.log > /tmp/file2.test `

real 0m4.862s
user 0m1.351s
sys 0m1.091s
[bharrington@berstuk tmp]$ time `sed -n ‘41,$p’ test.log > /tmp/file2.test `

real 0m4.606s
user 0m1.440s
sys 0m1.124s

Eclipse is a pretty common development too familiar to Java developers. I’m starting to get into the swing of Java development and decided that I would give it a try. I’d tried installing it in the past and it was a bit of a bugbear. Recently, I noticed that it is available in the yum repository for Fedora. I like package repositories so I figured i’d give it a shot. What followed was a path filled with some ups and downs. To make things easier for you, I’m writing up a cheatsheet as to how to get it running as fast as possible with as little headache as possible.

First and foremost, you’ll want to install the following packages in Fedora with yum:

yum install eclipse-platform eclipse-emf-xsd-sdk eclipse-emf-sdo eclipse-ecj eclipse-emf-xsd eclipse-jdt eclipse-emf-sdo-sdk eclipse-rcp eclipse-emf eclipse-gef eclipse-subclipse eclipse-phpeclipse

That will do much of the heavy lifting and should get you going with doing PHP, Java, etc development.

Your next step (if you dare) will be to use Eclipse’s “update manager”. Now the update manager in eclipse is not nearly as friendly as yum, apt, etc. It does the work of telling you “yo, you’re missing XXX dependency” but not much else. You’re going to have to dig, and it’s not always friendly. Googling to find the package name (and sometimes package name + eclipse & “update manager”) will normally give you some pretty good links to add to the update manager.

To get you started, here’s a copy of my exported sites. You can easily import this by saving it as an XML file and then using the “import sites” utility in the update manager.

Eclipse Bookmarks

Good luck!

Maybe some of you folks got a totally sweet camcorder for X-mas. If so, check this out. I’m a big fan.

$14 Steadycam The Poor Mans Steadicam

Hope santy claws left fun things for y`all.

After years of speculation and rumors that were touched off almost immediately following the release of The Lord of the Rings: The Return of the King, New Line Cinema and MGM Pictures have announced that a two-film version of The Hobbit (that’s right — TWO parts!!) will be happening, with Peter Jackson at the helm. Early on, there were no direct (public) plans to make The Hobbit. Then, there was acknowledged interest by all parties but Jackson and New Line got into a messy legal dispute about profits owed from the previous Lord of the Rings movies. Now, the two sides have apparently settled their differences and the green light has been given. However, Jackson has stated that he does not want to direct the films because he has obligations to other projects that would keep The Hobbit from being released in time for its initial 2010 projection. This has led to early speculation of Guillermo del Toro (Hellboy, Pan’s Labyrinth) and Sam Raimi (Evil Dead/Army of Darkness and Spiderman trilogies) as possible directorial candidates.

Of course, the top online news/blog/rumor source for all things LOTR-related during the fervor and excitement for the original movies, theonering.net, is buzzing once again at this news after a long period of relative dormancy. Check it out for all the latest!

The other day, I had a chance to see Celebration at 2640 in Baltimore… if you get a chance to see this band, go. It’s most of the members of another Baltimore band, Love Life, and they were very good live… Katrina, the singer, has this booming voice and presence, and the crowd was totally swept away… by the end it became a sea of dancing and arm waving. a very good time indeed.

Wow,

Let me say it’s been a while since I picked up a good piece of dystopian, tech savvy literature. Jeff Somers hit the nail on the head with his newest work “The Electric Church“.

Before I go into more details, here’s the closest thing to what I can think happened:

• Somers broke into William Gibson’s house, stolen his pens
• Next, broken into the storage space of Sterling and found an unused ream of copy paper
• Frantically channeled the spirt of these two men into a brutalized amalgamation of cyber-punk-esque madness

Now to convert you to the church:

The Electric Church(TEC) is based in a near future/post-apocalypse based world where crime is the standard and every man has his price. Thieves, gunners, and techies wade in the cesspool of civilization left by the unification of the system. Constantly hiding from the bootheels of the system security force, Avery Cates is a man who knows his price. A gunner for hire he has become adept at separating the politics from his work. Everyone has done something bad in their lives, it’s just a matter of whether the price is high enough to make you pay. With the rising tide of “The Electric Church” controlled by it’s founder & high priest Dennis Squalor, its the fastest growing religion. Monks on ever street corner promise eternal life and ‘an endless trail of sunsets’. But what happens when you become a pawn between the System and The Electric Church?

All in all, the book was written superbly. The language is not pretentious. In fact, the vocabulary, though small, is quickly established and maintained. Somers goes into great detail about how poor the quality of the bathtub gin is. I could have done well with him mentioning it only about half as many times as he did, but that’s his style. He takes an idea and runs with it and never hesitates to reuse a choice turn of phrase.

After reading the book, I must say I became even more of a fan. At the end of the book there was a link to a well made website: http://www.the-electric-church.com. Being the nerd that I am, I wondered what google may have indexed in the site which didn’t turn up much. A google search for sites which linked to it was much more fruitful and turned up a couple of character based blogs from the novel. Finding these was a nice touch. While in general, they exist more to help the main sites google page rank, it was awesome to see someone put in that kind of time into easter eggs, especially for a book.

So it looks like we’re going to be rolling out our new storage array soon. I’m going to be buying the hardware this week. It’s going to be a 2TB monster which we’re setting up to expand all the way to 14TB. If that doesn’t keep our backups in good working order, ain’t nothing going to keep our backups in good working order!

Sunday I got a chance to photograph one of my favorite MCs, Philly’s Amanda Blank performing with Chicago DJ team Flosstradamus. Flosstradamus has been getting noticed alot lately (I also shot them earlier this year for URB magazine) (press that is richly deserved because they bring it on the dancefloor). Blank is also getting well known for guesting on a variety of Spank Rock and Plastic Little songs, but now she is ramping up for her debut album. I was a little concerned because some of the advance tracks that have been released sounded pretty poppy, but live it was stellar.

Click here for photos from the show!

I recently purchased my first HDTV and found the process to be agonizing. The options seem endless, and the available information is simply staggering. 720p v 1080p? Plasma v LCD? Contrast ratio? Response time? Why is there a Sharp LC-42D62U model and a LC-42D64U model?? What’s the difference? Why are there 20 different Sony 46″ LCD models?! The more I read, the more questions I had rather than less. Everyone seemed to say something different.

I initially thought I wanted DLP over LCD or plasma because they offer the biggest picture bang for the buck. The only drawback seemed to be that the panel is not 3″ wide and can’t be hung on a wall, something I don’t care about at all. But the real main drawback for DLP is the viewing angle:  it is extremely narrow. If you move to either side, or stand up and look down at a DLP screen, the picture fades noticably, and this wouldn’t fly in my poorly-arranged living room. So, that’s out. LCDs have recently come down in price and up in size to compete with plasmas. I narrowed down my search to just LCDs based on the (somewhat antiquated but still valid) fear of burn-in with plasma: fixed images like sports or stock tickers eventually leave a burned-in image on the screen. This happens a LOT less on newer plasma models (apparently) but still is a possibility. So LCD it was.

Wired had a decent summary of the difference between 720p and 1080p. Basically, the human eye can’t tell the difference between the two on a screen smaller than about 50″, depending on how far away you are from the TV. On top of that, the number of true 1080p sources are very few right now. Blu-ray and HD-DVD, XBox 360 and I think PS3 are the only ones; no network broadcasts in anything higher than 720p or 1080i (a compromised 1080 resolution that has its drawbacks - read that article!) and I’ve read that no networks have any plans to switch to 1080p for the foreseeable future. That said, I still ended up buying a 1080p TV because, as one of the hot buzzword-ish specs for TVs that are selling right now, a lot of the accompanying specs are higher on the 1080p TVs, like response time (which is important for fast-moving images like sports) and contrast ratio and black levels.

I got sucked into the world of AVS forums and attempted to sift through the 10,000+ post threads on each popular model. Enter at your own risk! The most I took away from them was a general overall impression of the quality of various models, rather than a definitive solid choice. A lot of the nerds talk about specs and problems that the average person would never even notice. There is definitely useful info to be extracted, but it takes a lot of time and patience to read through it all.

cnet.com is of course the standard for electronics reviews, and I used it as well. The reviews are more accessible than the avsformums but still nitpicky on a level that doesn’t matter to the average consumer. The site helped me determine, though, that Samsung and Sharp seem to be the best bang-for-the-buck LCD. Pioneer and Sony seem to be the best overall, but you’re going to pay a pretty penny for them. Fairly certain that I could never tell the difference, I dismissed them as cost-prohibitive. After reading enough favorable reviews and seeing a couple of my friend’s Sharp Aquoses (what’s the plural of Aquos?), I opted for the 42″ LC-42D64U (LC-42D62U is a slightly older model with apparently rampant banding issues - one thing the avsforums.com nerds scared me away from) and am very happy with it. It gives a beautiful picture from my comcast digital HD box. My only regret is that I didn’t spring for the 46″ (another $700 didn’t seem worth it). My couch is 7′ away from the TV and I didn’t think that was too far for the 42″. When I sit closer I do notice some imperfections in the picture so I am definitely happy overall.

Good luck and happy hunting!