An elephant, a drunken man and a cron job

An elephant, a drunken man and a cron job

It was a serene evening one summer day in 2010. Sony, Sherin, Nandu and I sat chit chatting in an estate tucked away a few kilometers into the woods of Munnar. A much needed rest after a long 5 hour drive from our startup office. A bottle of Old Monk rested on a makeshift table in the little hut we were sitting around. Tempting as it may, the bottle had to wait.

...

A few days ago, Sherin had done some major refactoring to one of our billing scripts to make it run faster and more efficiently. See, at that time this startup of ours ran a very successful product catering to a big telecom's customers. The system worked on a daily subscription model which meant that we needed to charge the whole of our customer base every single day. It was the bread and butter of the company and any failures even for a few minutes had big consequences on our finances. The original program that ran this was facing difficulties keeping up with our growing user-base and there were days when we failed to complete a full cycle. This required a better system, but had to be shipped fast - and that was the quick & dirty script that Sherin came up with. It was not perfectly stable, had some quirks and issues which meant that it had to be run and monitored manually every night. That gave us scope to react to errors and outages, so that we don't end up losing revenues.

There was one small problem though - it had to be run on our production servers - and only three of us had access to the system. Nandu, Sherin & I would take turns to run and monitor the script.

...

The beauty of the estate we were in was that it was an escape from the world. A perfect place to chill and relax, but that also meant no cellular coverage. Our sole link to the outside world was an old landline telephone. In fact the only other soul apart from us in the 30 acre estate was an estate manager.

The estate manager soon joined our spot entertaining us with colorful stories of the secluded life in and around. It all was joyful till the moment we told him that we needed to head to town at night to catch some network to run the little billing script of ours. He gave us a slightly startled look and seemingly was puzzled by our silliness. It was then he told us about the elephant. With hearts in our mouth we listened to his story of a rogue wild elephant ruling the road to Munnar at night. Apparently he trampled and killed a motorbike rider just a few days back.

The silence was broken by Sony excaliming - "But we are going to lose 4 Lakhs! It's only a 30 minute drive, what could go wrong?" And lo, the decision was made - we were going to take one for the team. I was about to mumble, "Technically, only Sherin needs to go right?". I swallowed those words quickly and nervously reiterated to myself that teamwork is everything in a startup.

...

It was already quarter to midnight and the script was taking way longer than usual. The trip to the town was thankfully uneventful, but I could feel our bravery slowly melting away as the thought of the return trip in pitch darkness lingered. With all of us crammed in the car, Sherin was staring away at his terminal watching lines of logs scroll by. And finally the much awaited four letters arrived on the screen - DONE.

The sense of accomplishment infused new courage and positivity in us. But alas, it was to be short lived. At around half way through we heard a big thump. Everyone stared at each other. Can it be?

Well, it was. Our tyre had punctured.

The company car that we were in, an old Baleno, was ill maintained to say the least. But to our surprise it did have a decent spare tyre, but unsurprisingly there were no tools in the trunk to help us change it. There we were! The triumphant keyboard warriors in the middle of an elephant infested desolate forest road, sweating under the moonlight. With unexpressed fear we walked around with our mobiles to catch a signal - albeit also taking care not to stray more than a few paces from the car, just in case we need to run for cover. Why we thought that paltry metal box of a car would stave us off from the attack of a wild elephant, I would never know.

Those minutes felt like hours. You see, black rocks when lit under moonlight looks exactly like an elephant's butt - and Munnar was strewn with mighty black rocks. Someone would suddenly spot one and let out a quiet shriek. We had contemplated walking the 5 kilometers to the estate, but dismissed that silliness quite quickly, especially wary of how many of those rocks we'll see on our way.

And then another thump. Can it be?

Well, it was. A car!

A pair of headlights struck our eyes glaringly as a van writhed painfully towards us negotiating the potholes. Screeching to a stop, a man dressed in a spotless white shirt and   mundu  and reeking of cheap brandy, stepped out from the passenger side and asked with a slurred tongue "ഇതെന്താ കുട്ടികളെ പറ്റിയെ?" (   What's wrong boys?  ). The story was told and the van had the tools. We took the tyre jack and spanner and was about to test our mechanical skills - when the burly man wrestled the spanner out of my hand and asked with a bit of sarcasm. "ഇതൊക്കെ നിന്നെ കൊണ്ട് പറ്റുവോ?"   (Will you be able to do all this?).  Before we knew what was happening, driven by either the goodness in him or the brandy in his gut or maybe both, he proceeded and changed the tyre faster than I've ever seen it done before. He then stood up straight, muttered that all is done and we're safe to go. Safe? But the elephant en route? "Follow the van!". He ordered, we obeyed. No elephants stopped us anywhere and we finally made it to the gate of the estate. The guy waved from the window as the van drove off to the distance. I am agnostic about angels, but if there's one I'm sure he wears a white   mundu  and loves brandy.

...

I perhaps sensed a hint of disappoint as the estate manager welcomed us back with a quip - "Ah! So you made it safe!". That Old Monk though was still waiting patiently on the table. It never had tasted sweeter ever.

...

That my friends is how we learned never to push ad-hoc code into production, to always set up a cron job and setup automated monitoring systems for critical parts of code.

A wild unknown elephant can sometimes teach you the fundamentals of software engineering. True story bro!

Tags 
Previous post
Goa - The last stop on the hippie trail
Next post