NodeJS + Cron: A series of attempts in getting rid of dangling processes

           ·

Short-lived scripts that sucked your resources for 1mins or at most 5mins - is probably fine. However, if it is recurring jobs in Cron and involves several tasks and at least an hour of processing - that’d be a different story.

I was working on a simple NodeJS-based cron script with only 3 requirements:

Each one of these however would involve crunching through a million rows of data.

If you take a closer look at the result of ps aux command, you’ll notice that there are couple of dangling processes waiting for some event to complete - of which technically the parent process doesn’t exists anymore! - the dates would tell you that some of those process had been there for about 9 days.

the script runs on top of AWS EC2 instance so literally we’re talking about:

CPU ticks, RAM usage, disk spins, etc == $$$

If left unattended this would turn into a zombie manufacturing plant. Anyhow, see the command output below.

$> ps aux:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      6934  0.0  0.0  59636   448 ?        S    Feb10   0:00 CRON
ubuntu    7005  0.0  0.0   4440   104 ?        Ss   Feb10   0:00 /bin/sh -c /home/ubuntu/crontools/cron-script-wrapper.sh > /dev/null 2>&1
ubuntu    7008  0.0  0.0  11112   180 ?        S    Feb10   0:00 /bin/bash /home/ubuntu/crontools/cron-script-wrapper.sh
ubuntu    7094  0.0  0.0   4444   108 ?        S    Feb10   0:00 sh -c npm run export-task && npm run transform-task && npm run load-task
ubuntu    7095  0.0  0.1 574624  5924 ?        Sl   Feb10   0:00 npm run export-task
ubuntu   15536  0.0  0.0   4440   104 ?        Ss   Feb18   0:00 /bin/sh -c /home/ubuntu/crontools/cron-script-wrapper.sh > /dev/null 2>&1
ubuntu   15557  0.0  0.0  11112   176 ?        S    Feb18   0:00 /bin/bash /home/ubuntu/crontools/cron-script-wrapper.sh
ubuntu   19398  0.0  0.0   4440   100 ?        Ss   Feb11   0:00 /bin/sh -c /home/ubuntu/crontools/cron-script-wrapper.sh > /dev/null 2>&1
ubuntu   19399  0.0  0.0  11112   192 ?        S    Feb11   0:00 /bin/bash /home/ubuntu/crontools/cron-script-wrapper.sh
ubuntu   19410  0.0  0.0   4444   108 ?        S    Feb11   0:00 sh -c npm run export-task && npm run transform-task && npm run load-task
ubuntu   19421  0.0  0.0   4444   108 ?        S    Feb11   0:00 sh -c node ./js/cron-script.js export-task
ubuntu   19422  0.0  1.1 839088 44164 ?        Sl   Feb11   0:01 node ./js/cron-script.js export-task
ubuntu   25683  0.0  0.0   4440   100 ?        Ss   Feb14   0:00 /bin/sh -c /home/ubuntu/crontools/cron-script-wrapper.sh > /dev/null 2>&1
ubuntu   25684  0.0  0.0  11112   188 ?        S    Feb14   0:00 /bin/bash /home/ubuntu/crontools/cron-script-wrapper.sh
ubuntu   25712  0.0  0.0   4444   104 ?        S    Feb14   0:00 sh -c npm run export-task && npm run transform-task && npm run load-task
ubuntu   25723  0.0  0.0   4444   104 ?        S    Feb14   0:00 sh -c node ./js/cron-script.js export-task
ubuntu   25724  0.0  2.6 964268 102912 ?       Sl   Feb14   0:02 node ./js/cron-script.js export-task
ubuntu   26093  0.0  0.0   4440   100 ?        Ss   Feb14   0:00 /bin/sh -c /home/ubuntu/crontools/cron-script-wrapper.sh > /dev/null 2>&1
ubuntu   26094  0.0  0.0  11112   188 ?        S    Feb14   0:00 /bin/bash /home/ubuntu/crontools/cron-script-wrapper.sh
ubuntu   26107  0.0  0.0   4444   108 ?        S    Feb14   0:00 sh -c npm run export-task && npm run transform-task && npm run load-task
ubuntu   26118  0.0  0.0   4444   108 ?        S    Feb14   0:00 sh -c node ./js/cron-script.js export-task
ubuntu   26119  0.0  1.5 929424 60320 ?        Sl   Feb14   0:01 node ./js/cron-script.js export-task
ubuntu   26511  0.0  0.0   4440   104 ?        Ss   Feb09   0:00 /bin/sh -c /home/ubuntu/crontools/cron-script-wrapper.sh > /dev/null 2>&1
ubuntu   26512  0.0  0.0  11112   188 ?        S    Feb09   0:00 /bin/bash /home/ubuntu/crontools/cron-script-wrapper.sh
ubuntu   26523  0.0  0.0   4444   108 ?        S    Feb09   0:00 sh -c npm run export-task && npm run transform-task && npm run load-task
ubuntu   26534  0.0  0.0   4444   104 ?        S    Feb09   0:00 sh -c node ./js/cron-script.js export-task
ubuntu   26535  0.0  0.7 601220 28212 ?        Sl   Feb09   0:01 node ./js/cron-script.js export-task

On the day when the script was loaded onto crontab and didn’t get was I was hoping for, my gut tells me that I f***ed up Cron: bad syntax or wasn’t throwing the logs properly, why? you asked! - having passed the “Works on My Machine” certification program I could run the script manually at sucess rate of 100%.

¯\_(ツ)_/¯

E-V-E-R-Y-T-H-I-N-G WORKS!

Unsuccessful Attempts

1st attempt:

* * * * * * cd /to/script/folder && npm run export-task && npm run transform-task && npm run load-task 2>&1

2nd attempt:

* * * * * * cd /to/script/folder && npm run-task-in-sequence 2>&1

3rd attempt:

* * * * * * cd /to/script/folder && ./script-wrapper.sh 2>&1

script-wrapper.sh

#!/bin/bash

npm run export-task
npm run transform-task
npm run load-task

exit 0

Successful Attempts

4th attempt:

* * * * * * cd /to/script/folder && ./script-wrapper.sh 2>&1

script-wrapper.sh

#!/bin/bash

node ./cron-script.js export-task
node ./cron-script.js transform-task
node ./cron-script.js load-task

exit 0

Not relying on running arbitrary commands in npm did fixed the issue.

Notes

References

comments powered by Disqus