Blizzard of 2008
I was almost four years old during the Blizzard of '78. I don't remember much, other than playing in the snow in our front yard, and mom and dad taking turns using the snow shoes they had once purchased as a souvenir.
We had another blizzard last weekend:
The snow started falling Friday around noon, and didn't stop until Sunday morning. I never bothered to check the snow depth in our yard, other than to confirm that it was about calf-high on me in most places, but I've heard reports between 15 and 20 inches around town. I shoveled the walk Friday evening, and laid down a scattering of salt, to help stave off additional work Saturday morning. Our neighbor was doing the same, and we shared warm conversation between breaks in our labor.
Saturday morning the kids were up and frolicking in the snow almost before I reached the bottom of the stairs. I suited up for the weather, and joined them outside. I shoveled a fresh path along our sidewalk, and neglected completely to dig out the cars from the driveway, before ambulating over to the neighbor's across the street. A collection of kids (most of them kids and grandkids of the neighbor) were collecting at the top of the hill, preparing to turn it into an Olympic-caliber bobsled run!
The kids spent several hours sledding on Saturday, and again on Sunday. A good time was had by all. The entire weekend reminded me why I'm so glad to live in the neighborhood we do, as kids played together, adults chit-chatted, and neighbors helped one another extricate cars, remove snow from porch roofs, and in general helped one another.
Jerk
I walked to the snack shop yesterday afternoon before the final conference call of the day. I was third or fourth in line, and the customer at the register appeared to be having trouble with their debit or credit card. It looked like she was only trying to purchase a soda, and the clerk informed her that she thought her card had been declined (the transaction failed twice with a communication error, which another employee claimed indicated a declined purchase). The girl stepped aside, and began calling someone on her cell phone -- I presume it was her bank or card company.
The line moved expeditiously forward, and before long I was standing at the register. The girl was still standing to the side, and still on the phone, though she didn't seem to be speaking to anyone. I asked the clerk if the girl was only trying to buy a soda -- which was still sitting on the counter -- and when I received an affirmative response I indicated that I would purchase it for her. The clerk looked surprised, but I confirmed my intention to purchase the soda on behalf of the young woman on the phone.
I had hoped that the girl would hear this, so that she would know that she could take her soda and leave, to continue her negotiations with her bank in a more comfortable, and private, environment. To my dismay, as I paid, the girl walked toward the back of the line, leaving the now-paid-for soda on the counter. When my transaction was complete, I walked back to her and said "Excuse me, miss. I purchased that soda for you."
It was my intention to make it clear to her that she didn't have to wait around in the snack shop on her cell phone -- that she was, in fact, free to leave. She looked at me for a moment, and then said, simply, "Okay." I responded in kind and walked out.
On the way back to my office, I realized that the girl probably thought I was a big jerk, proudly declaring my good deed in a pompous show, expecting great thanks from her. And realizing that, I really do feel like a jerk, now.
Microsoft Windows Unified Data Storage Server 2003
Last summer, my employer purchased a Hewlett Packard DL585 server running Microsoft Windows Unified Data Storage Server 2003. This server was selected and purchased to provide unified home directories to all our users across Windows and GNU/Linux machines. Windows users would connect via CIFS, and GNU/Linux users would connect via NFS. When a user logs in to a GNU/Linux machine, they have the exact same home directory as when they log in from a Windows XP workstation.
I followed these instructions for integrating our GNU/Linux server into the Active Directory domain. The server is a computation server, to which our users connect via ssh. We had some trouble initially getting Active Directory authentication working. The VAR from whom we purchased the file server sent out a technician who spent two days on-site with us, one of which was spent almost entirely (on hold) on the phone with Microsoft support. At long last, we finally got all the pieces of the puzzle assembled, and users could authenticate to the GNU/Linux server using their Active Directory credentials.
After a few weeks, I had reports that a user lost some of their files while using the GNU/Linux server. It took me a couple days of investigation to finally figure out what the problem was. When I did, I was horrified. You see, all of our users have a 20GB disk quota, meaning that they can save only 20GB of data into their home directory. The file server enforces this quota, and we had no problems with this configuration for our Windows workstations. Remember, Windows workstations communicate with the file server via the CIFS protocol. What I discovered was that the file server did not seem to be enforcing the quota for GNU/Linux machines connecting via the NFS protocol. At first, it looked like a user could exceed their quota -- and indeed, they could for a short period of time, provided they stayed logged in. If the user logged out, or after a few minutes of inactivity, the file server would truncate to zero bytes those files that had exceed the quota. What was worse, a file opened for writing before the quota was reached would also be truncated if the size of that file grew to cause the user to reach their quota. What we had was silent data loss. The purpose of this server is for long-running batch computations, some of which generate enormous amounts of data: the possibility that someone's research job running for days or weeks could silently lose its dataset was catastrophically bad.
We had a few problems getting HP support engaged properly, mostly due to the front-line support folks not really understanding my problem report, and dispatching me to the wrong group. After that was cleared up, we finally connected with a couple of level 3 support engineers who took complete ownership of the problem. They issued a few hotfixes for us (one to resolve bluescreens when you enable logging for NFS on the file server -- yeah, that was a fun day), and walked us through a number of diagnostic steps. In the end, we collected several gigs worth of packet traces and memory dumps. Along the way, we identified a few more problems, for which additional hotfixes were issued. After a few weeks, the HP engineer confirmed for me that they were able to reproduce the quota problem for which I had originally called: the Microsoft NFS server code did not enforce quotas properly, resulting in silent data loss.
The issue was escalated upstream to Microsoft on November 6, 2007. As of today, Microsoft has confirmed that they can reproduce the problem; but they're telling HP that they will not commit to a specific date by which the problem will be fixed. Microsoft suggests that we work around the problem by using the "sync" mount option for our NFS clients. Yes, that works, but imposes a non-trivial performance penalty, which can be a real problem for the intended use of this server: lots of grad students crunching numbers and spooling data sets to disk for analysis.
I've spent much of the last 24 hours in meetings with my coworkers and on conference calls with our VAR, HP's escalation support manager, the level 3 HP engineer who owns this issue, HP pre-sales engineers, and HP product engineers trying to figure out how Hewlett Packard can resolve this situation for us. It's clear they sold us something that doesn't do what we all thought it would -- and should -- do. I'm relieved that HP is involved to this level to make things right. It's a shame, though, that the real problem can't be fixed upstream and pushed out to us. Instead, we're looking at a complete overhaul of our storage solution, and a substantial new investment of time and energy.
We have two options on the table: an HP MSA or EVA series SAN with clustered GNU/Linux servers running Polyserve clustered file system for "Enterprise File System Clustered Gateway", or one of the SAN backends with two independent servers running Windows and GNU/Linux for CIFS and NFS shares, respectively. The former option preserves our unified home directory configuration, while the latter unambiguously connects the server operating system with the same client system. The drawback to the separate server solution, though, is that users would have two separate home directories, and double the quota we had originally intended to give them (we're ruling out the notion that GNU/Linux users split their quota capacity between servers, or that only folks who ask for it get a Linux account, or other such management nightmares). We don't yet have final pricing on either option, because we're looking at a buy-back situation for our current hardware, plus hopefully some modest discount to make up for the time and energy we've spent fighting this problem for so long. Price may well be the deciding factor.
What would you do, if you had to support 1,000+ students and about 100 faculty and staff?
Daily Commute
Last week, in preparation for my walk to work, I strapped some additional traction onto my shoes in order to help me traverse the ice and packed snow. As usual, I donned my snow pants, heavy winter coat, gloves, and a hat. I walked hunched over, to keep the warmth of my breath close to my chin and exposed cheeks.
This morning I donned only a light jacket, and even that proved a bit stifling as a result of the strong pace I set for myself. I walked with my head held high, and enjoyed the damp coolness of the morning. Without an audiobook to enjoy, I listened instead to the sounds of the songbirds, of which there was no shortage.
Last week my eyes looked down at the dirty white ice and gray snow. This morning my eyes scanned side to side looking at the green grass and the brown earth that was now exposed. I won't be so bold as to say "Spring has sprung," but I am glad that the daily commute should only get more comfortable from here on out.
Movies
While looking to clean up my laptop's hard drive yesterday, I saw that I had downloaded a number of full-length features from Miro, but had never watched them. I transferred them to my MythTV computer with the intent of watching them on the television.
This afternoon I watched The Last Man On Earth, the original movie of which I Am Legend is a modern re-telling. The original was interesting, and I wish now that I had watched it prior to seeing the latter, which I didn't particularly like. I tried to get the kids to watch it with me this afternoon, but they only watched it in small snippets, convinced that it would be scary despite my claims that movies made back then didn't employ the "jump out and scare you" tactics of modern "horror" movies. I've seen Vincent Price in a handful of other films, but I'd really like to explore his works in greater depth now: his performance was very commanding in "The Last Man on Earth."
I made Carina watch The Fountain with me awhile back, and she liked it considerably more than I did. I watched it primarily because of Hugh Jackman, with whom I continue to be impressed, but also because the Netflix description of The Fountain suggested it was a slightly different story than that which was actually told. In particular, I kept waiting for "the 26th century astronaut." When the movie was over I explained to Carina why I was so disappointed in an otherwise acceptable movie. She got a hearty laugh at my expense.
To make up for the lack of astronauts in The Fountain, I next watched Mission to Mars. I generally like Brian de Palma films (I especially liked Raising Cain), and this movie did not disappoint. The star-studded cast was extremely enjoyable to watch, and the three acts of the story each worked to produce a cohesive whole. It was vastly superior to Red Planet and Ghosts of Mars in terms of Mars movies. The special features on the DVD provided some extremely interesting insight into the production of the movie, too.
In an effort to share some of myself with my kids, I've been adding some of my favorite childhood movies to their Netflix queue, and watching them with the girls. They both really enjoyed The Black Hole, which was a tremendous delight for me. During the scene where Maximillian approaches Anthony Perkins' character, Tayler said softly to herself "Oh shit!" which made me cover my mouth and bite my tongue so as not to laugh out loud and thus ruin the feeling of the moment for her. While not thrilled with her language choice, I was pleased beyond words that she could get so into the movie. I had forgotten how tense some of the scenes were in that movie, and it really got me thinking about what constitutes "kids movies" today. We recently watched The Spiderwick Chronicles as a family, and I think it's the closest thing I've seen to a good kid-friendly scary movie in a long time. I hope more films like it are made for adolescents.
Tonight I put on Cool Runnings, a movie in which I have always taken something of a guilty pleasure. It's not a particularly good movie, but it is fun and tells a sweet story. I thought that both of the kids would enjoy rooting for the underdogs, and could share in the value of the lesson taught. Unfortunately, we started late in the evening, and since we all stayed up late all weekend long, Tayler simply couldn't keep her eyes open. She went to bed about a third of the way into the movie. Tyler stuck it out with me, though, and she clapped her hands at the end, as the Jamaicans walked their sled across the finish line. I'm confident that Tyler will want to watch it again, and I know Tayler will enjoy it when she watches it.
Earlier this evening, though, I broke out a few DVDs of home movies I made of the twins from 2003 and 2004. It was so much fun to watch! The girls were so little, and their voices so sweet. They were both fond of putting on "shows" for Carina and I, although they never bothered to prepare or even discuss their plans with one another before doing so, so their "shows" usually ended up being impromptu attempts at hogging the camera from one another. The girls laughed out loud as they watched their younger selves dance and wiggle and read from books and lead the camera on a tour of our house. I asked them if I could put a few of these on YouTube for the whole world to enjoy, but I was resolutely denied.
I need to remember to use our video camera to keep recording the kids: it's such a delight to see those little moving snapshots of history.
