Tuesday, April 7, 2009

Cron does not work after reboot - Solved

I have been quite bewildered with the unreliable behavior of the cron jobs I set up for my project recently at work. I needed data to be imported periodically from an FTP server and then process the data and put into my application's database.

Every time I would open the file to see if there's something I have put in wrong there. I would make some 'trial and error' changes and it would work beautifully. However, next morning when I returned to office and saw the logs of my application, there wouldn't be any sign of the cron jobs having run for some time.

I googled the problem. Everyone seemed to outright reject the idea that cron can ever fail. They explained different things that might be wrong - environment variables, wrong paths etc. etc. Cron was infallible, it seemed. Then why was I having a problem.

I contacted the genius who goes by the name of Kartik Reddy Thum. He was equally bewildered. "Cron is supposed to be quite reliable", he reasoned too. He suggested I move whatever I was trying to run to a shell script, put in some logging and call the script from the cron tab. Earlier I was calling the curl command directly from inside the cron tab. I moved the curl call to a shell script and called the script from cron. After this, cron worked for some time. Next day, however, when I examined the logs, it had stopped after a while!

One thing which I realized was that due to the daily hour-long power cuts prevalent in the city these days, the server that ran my application would undergo an improper shutdown and then come back up when the power returned. Yes, I can hear you yell - "How can you not have power backup for your server?" Well, we do have power back up but its only for 10 minutes whereas the government of the day has decided we need atleast an hour long power cut. (Now, last Sunday we had a two hour cut and that too not at the usual time - but that's a topic for another post!)

So, I knew that the server was shutting down ungracefully. Was this, in some way, responsible for this strange behavior? However I did not see why that should be an issue. When the server starts up again, cron should kick in quite effortlessly.

I decided to google again but added the keyword "reboot" to the other words. Bingo! There was the problem. Cron did not kick in after a reboot. For some reason, you had to edit the cron tab file after a server reboot for it to function. Many people had a similar problem and they simulate an edit for the cron to work after a reboot.

Beware - a simple "touch " does not work. You have to do a crontab -e, make some edit and then save it for this to work.

On the way to solving this, I figured out where the cron tabs are located on a Mac. They are not in /etc/crontab as widely thought. They are in /var/at/tabs. One file for each user who has ever done crontab -e and saved something. Also, if the root user is enabled, this folder belongs to root and you cannot view its contents as non-root.


Kartik said...

Cool :) ! So, now we know, thanks!

Don said...

Ha! That's happened to me a few months ago on my Airwave servers; cron wouldn't run an executable perl script because the cron root user wasn't inheriting root's environment variables as expected. I ended up writing a small shell script to set the variables and run the perl script. Admittedly an ugly fix but problem solved. 8-)