You might want to check the source of existing Webmap Solutions (OpenLayers, Leaflet, ...) or GIS / OSM / .... frameworks and get ideas for your architecture.
Usually you have modules for the following aspects:
- multi CRS / projection
- vector primitives
- vector/raster layers
- basemap protocols and providers
- routing, geocoding, .... extras
- ...