OSGi is definitely the right technology if you want to create a modular application that can be updated quickly without downtime.
You have a couple of choices:
You can start with an OSGi container like Apache Felix, and deploy bundles into that container for both your application and a webserver (most of them are based on Jetty). That way, everything is a bundle and can be updated if needed. If you're starting from scratch and have no constraints on existing servlet containers that you must use, this is the way to go.
If you have to keep working with your existing servlet container, you can deploy a WAR that embeds an OSGi container and the OSGi container will load all the bundles that make up your application. In this case, you can deploy a special "bridge" bundle that will register Servlets and Filters with the "host" servlet container.
Starting with Glassfish is a bit like the first option, since Glassfish itself is built on Felix. You just get a lot of "extra" bundles pre-installed in your application. If you use 80% of those anyway, that is probably worth it. If you hardly use any of Glassfish's (JavaEE) features in your application, you're probably better off starting with just Felix and selecting components there.