Short: you should always worry about server side validation.
Long: Javascript cannot easily be manipulated within the app, but rest assured that the frontend is never safe. In fact, the Javascript and HTML/CSS of a Phonegap application can easily be extracted and they could rebuild your app with just a few differences.
In addition, anybody could fire calls to your backend services (from self-built front-ends, REST clients or anything really), so be very sure that your services are safe and that your data is validated..!