Marvin Framework provides feature extensibility through plug-in interface. Basically, it allow you to implement an image processing algorithm as a Java class that can be loaded dynamically on your application.
Regarding your idea, consider a Java application combining an image editing software and a source code editor. After writing the image processing algorithm, the user clicks on "Execute". The application:
- saves the source code as .java file.
- call the Java compiter to generate the .class
- Load the new class dynamically into the application through Java Reflection
It is basically what MarvinEditor does to let developers to add new plug-ins to the application. Take a look at Marvin Prototyping Environment. It's similar to your idea.