How to avoid churn when you are defining a schema while others are trying to program against it

Question

I'm in charge of defining an XML schema for some data; the plan is to have various systems (all under our control) upload XML files that match this schema to a central server for processing. I don't have much knowledge of the design of these other systems, so I'm working collaboratively with those system experts to make this XML schema reasonable to both consume (on our side) and produce (on their side).

The schema is not entirely clear-cut, and I'm defining the data we want to collect at the same time as defining the schema. There are a million ways to do it so there's a definite "do something, think about it, talk to so-and-so, refactor, repeat" process in place.

All of that said, we have an unfortunate situation where myself and a few others are defining things and a few other developers are implementing it. I'd like to write some code but haven't found the time. This leads to the frustrating cycle whereby the design changes in the ivory tower and then the guys writing the code have to go and change it again and again. Avoiding details for various reasons, the developers writing the implementation aren't in a position to effectively define the requirements of the schema and the data we must collect.

Should I be the one writing and playing with this stuff as I figure out how it should be done? This would be ideal to me, but we're under pressure to get things done and it's quicker to have a few people working on it rather than just me.

Do we have too many cooks in the kitchen with the XML file providers influencing the schema? Ease of XML file creation should be a consideration of the schema, right?

Is there a better way to solve this problem, or is churn a necessary evil when you're trying to fly the plane as you build it?

What is the correct development process to apply here which will minimize wasted time?

Answer 1

I am not sure if it is a way to definitively reduce churn and how applicable it would be to your current situation, but these are some things I would try to do -

Minimum Viable Product

I am not sure if this is fully relevant to your current situation since it looks like you are already in the middle of things but I am going to put it in anyway since it provides a context to some of the other points.

I would push to identify the bare minimum of data needed and try to build the smallest end to end integration possible first. Once that is done it becomes a lot easier to handle further changes as a lot of the initial questions around approach and risk have been addressed.

Tight communication loop

Since you have mentioned that you are not in a position to work on coding the modification to the systems or building a prototype yourself the next best thing is to work closely with the people who have to do this.

While building up the schema specification I would try to keep the communication loop as tight and small as possible. Identify a small team that can handle the project end to end and work closely with them (preferably in the same room).

If you have multiple teams for different systems I would identify certain folks from each team and work with them. If the teams are too small for this I would try to get a specific block of time to focus on the schema integration and build it in focussed sprints.

Build for easy modification/extension

Since you know this is a rapidly changing and sometimes ambiguous schema it is important to design the schema and the code to generate it such that they can be easily modified.

If this is done well, after the initial minimum viable product built out, it would be much easier for the developers to extend the system and reduce the churn time.

---

Answer 2

Add a version element or attribute up near the root of your xml. Each time you publish a new schema increment this version. Producers set the version and use the corresponding schema. Consumers read the version and apply the appropriate shredding. This works best if the position of the version tag doesn't change between versions.