Execute Python (selenium) script in crontab

Question 1

A quick guess, your job fails to run because it requires an X session. You should setup your test script to run in a headless mode.

update:
Your trace says exactly what I meant, firefox does not run without X, or a display if you want.

 The output was: Error: no display specified\n'

Save yourself more trouble, read the link I posted.

Question 2

As seen here, you need to give cron a display:

30 5 * * * export DISPLAY=:0; /home/me/good_morning.sh

Question 3

You can do it without crontab.

I found schedule modul: GIThub/schedule

Now your code should look like:

import schedule
import time
def job():
    import pickle
    from selenium import webdriver
    driver = webdriver.Firefox()
    driver.get('http://www.google.com')
    t=driver.current_url
    pickle.dump(t,open('noreal','wb'))
schedule.every(1).minutes.do(job)
while True:
    schedule.run_pending()
    time.sleep(1)

It is not best solution but it works.

Question 4

This issue has bugged me for near 6 hours.

So if I write /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py in terminal it is working fine.

But it doesn't work for crontab. Right?

Adding

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()

will not help.

What you need is to make a shell script and write the following -

#!/usr/bin/env bash

export DISPLAY=:0
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
nohup /usr/bin/python3 -u /root/apps/bridge/cinkselenium.py > /root/apps/bridge/cinkselenium.log &y

Now run the shell script from Crontab -

* * * * * /root/apps/bridge/cinkselenium.sh

This is what works properly for selenium in crontab without headache in DigitalOcean servers. The paths will be different based on your server and program location.

Question 5

Install Xvfb or Xephyr or Xvnc. Also install pyvirtualdisplay python module. This is for Ubuntu 14.04 and tested on 16.04 as well.

sudo apt-get install python-pip
sudo apt-get install xvfb xserver-xephyr vnc4server
sudo pip install pyvirtualdisplay
# optional
sudo apt-get install python-pil scrot
sudo pip install pyscreenshot

Now you can start Xvfb from your python script. It will start Xvfb when your script will be executed. Add the following lines to your code.

from pyvirtualdisplay import Display
display = Display(visible=0, size=(1024, 768))
display.start()

Here is the example code.

from pyvirtualdisplay import Display
from selenium import webdriver

display = Display(visible=0, size=(1024, 768))
display.start()
browser = webdriver.Firefox()
browser.get('http://www.google.com')
print browser.title
browser.quit()

display.stop()

Now you can mention the program in your crontab. But add the path of ipython instead of ipython only.

0 * * * * /usr/bin/python /home/usr/Data/progs/cron_test.py build