How do I manage multi-projects dependency (Directed acyclic path) with IBM RAD ant?

Question

I am working on an ant script to build java prjects developed with IBM RAD 7.5.

The an script is calling IBM RAD ant extenstion API. I am using Task to load the project set file(*.psf) into the memory, and calling Task to compile the projects listed in the projectSetImport.

The problem is the projects listed in psf file is not ordered by project dependency, when compiles, it fails because the depency is incorrect.

Is there any API or method to manage the dependency automatically? the psf files Iam handling is quite big, with 200+ projects in each file and it is constanly changing(e.g. some projects get removed and some new projects added in each week)

here is a detailed description for the question: The project dependency is like: 1) project A depends on B and D. 2) project B depends on C 3) project E depends on F

A -> B -> C
A -> D
E-> F

The sample.psf file just list all projects:

A
B
C
D
E
F

loads sample.psf, which have a project list [A,B,C,D,E,F] build project list from the build fail at A, because A need B and D to be build first.

My current solution is to rebuild the sample.psf manually, e.g. sample.psf file:

C
B
D
A
F
E

but this is hard to maintain, because there are 200+ projects in a psf file and they are constanly changing.

One way to attack this issue is to write a parser to read the .project file for each project, the dependency projects are listed in "projects" tag. Then implement a Directed acyclic path algorithm to reorder the dependency. This approach might be over kill. This must be a common issue in teams build IBM java projects, is there a solution?

Answer 1

Finally, I wrote some python code to compute the dependency. I Listed the logic below:

1. read the psf file into an list, the psf file is a xml file, and the project name is in tag.
2. for each project in the list, go to project source code and read the .project file and .classpath file, these two files contains the dependency project. for .project file(xml), fetch the project name from tag, for .classpath file. fetch the line with attribute kind='src'
3. now you got [source]->[dependened_project_list], implement a Directed acyclic map.(see attached code)
4. load the [source]->[dependened_project] in to the AdjecentListDigraph, and call topoSort() to return the dependency.

5. generate a new ordered psf file.

   /////////////////////// dap_graph.py/////////////////////////////
   # -*- coding: utf-8 -*-
   

   '''Use directed acyclic path to calculate the dependency''' class Vertex: def init(self, name): self._name = name self.visited = True

   class InValidDigraphError(RuntimeError): def init(self, arg): self.args = arg

   class AdjecentListDigraph: '''represent a directed graph by adjacent list'''

   def __init__(self):
       '''use a table to store edges,
       the key is the vertex name, value is vertex list 
       '''
       self._edge_table = {}
       self._vertex_name_set = set()
   
   def __addVertex(self, vertex_name):
       self._vertex_name_set.add(vertex_name)
   
   def addEdge(self, start_vertex, end_vertex):
       if not self._edge_table.has_key(start_vertex._name):
           self._edge_table[start_vertex._name] = []
       self._edge_table[start_vertex._name].append(end_vertex)
       # populate vertex set
       self.__addVertex(start_vertex._name)
       self.__addVertex(end_vertex._name)
   
   def getNextLeaf(self, vertex_name_set, edge_table):
       '''pick up a vertex which has no end vertex. return vertex.name.
       algorithm:
       for v in vertex_set:
           get vertexes not in edge_table.keys()
           then get vertex whose end_vertex is empty  
       '''
   

   print 'TODO: validate this is a connected tree'

       leaf_set = vertex_name_set - set(edge_table.keys())
       if len(leaf_set) == 0: 
           if len(edge_table) > 0:
               raise InValidDigraphError("Error: Cyclic directed graph")
       else:
           vertex_name = leaf_set.pop()
           vertex_name_set.remove(vertex_name)
           # remove any occurrence of vertext_name in edge_table
           for key, vertex_list in edge_table.items():
               if vertex_name in vertex_list:
                   vertex_list.remove(vertex_name)
               # remove the vertex who has no end vertex from edge_table
               if len(vertex_list) == 0:
                   del edge_table[key]
           return vertex_name
   
   def topoSort(self):
       '''topological sort, return list of vertex. Throw error if it is 
       a cyclic graph'''
       sorted_vertex = []
       edge_table = self.dumpEdges()
       vertex_name_set = set(self.dumpVertexes())
   
       while len(vertex_name_set) > 0:
           next_vertex = self.getNextLeaf(vertex_name_set, edge_table)
           sorted_vertex.append(next_vertex)
       return sorted_vertex
   
   def dumpEdges(self):
       '''return the _edge_list for debugging'''
       edge_table = {}
       for key in self._edge_table:
           if not edge_table.has_key(key): 
               edge_table[key] = []
           edge_table[key] = [v._name for v in self._edge_table[key]]
       return edge_table
   
   def dumpVertexes(self):
       return self._vertex_name_set
   
   //////////////////////projects_loader.py///////////////////////
   

   -- coding: utf-8 --

   ''' This module will load dependencies from every projects from psf, and compute the directed acyclic path.

   Dependencies are loaded into a map structured as below: dependency_map{"project_A":set(A1,A2,A3), "A1:set(B1,B2,B3)}

   The algorithm is: 1) read 2) call readProjectDependency(project_name) ''' import os, xml.dom.minidom from utils.setting import configuration

   class ProjectsLoader:

   def __init__(self, application_name):
       self.dependency_map = {}
       self.source_dir = configuration.get('Build', 'base.dir')
       self.application_name = application_name
       self.src_filter_list = configuration.getCollection('psf',\
                                                           'src.filter.list')
   
   def loadDependenciesFromProjects(self, project_list):
       for project_name in project_list:
           self.readProjectDependency(project_name)
   
   def readProjectDependency(self, project_name):
       project_path = self.source_dir + '\\' + self.application_name + '\\'\
           + project_name
       project_file_path = os.path.join(project_path,'.project')
       projects_from_project_file = self.readProjectFile(project_file_path)
   
       classpath_file_path = os.path.join(project_path,'.classpath')
       projects_from_classpath_file = self.\
           readClasspathFile(classpath_file_path)
   
       projects = (projects_from_project_file | projects_from_classpath_file)
       if self.dependency_map.has_key(project_name):
           self.dependency_map[project_name] |= projects
       else:
           self.dependency_map[project_name] = projects
   
   def loadDependencyByProjectName(self, project_name):
       project_path = self.source_dir + '\\' + self.application_name + '\\'\
           + project_name
       project_file_path = os.path.join(project_path,'.project')
       projects_from_project_file = self.readProjectFile(project_file_path)
   
       classpath_file_path = os.path.join(project_path,'.classpath')
       projects_from_classpath_file = self.\
           readClasspathFile(classpath_file_path)
   
       projects = list(set(projects_from_project_file\
                            + projects_from_classpath_file))
       self.dependency_map[project_name] = projects
       for project in projects:
           self.loadDependencyByProjectName(project)
   
   def readProjectFile(self, project_file_path):
       DOMTree = xml.dom.minidom.parse(project_file_path)
       projects = DOMTree.documentElement.getElementsByTagName('project')
       return set([project.childNodes[0].data for project in projects])
   
   def readClasspathFile(self, classpath_file_path):
       dependency_projects = set([])
       if os.path.isfile(classpath_file_path):
           DOMTree = xml.dom.minidom.parse(classpath_file_path)
           projects = DOMTree.documentElement.\
               getElementsByTagName('classpathentry')
           for project in projects:
               if project.hasAttribute('kind') and project.getAttribute\
                   ('kind') == 'src' and project.hasAttribute('path') and \
                   project.getAttribute('path') not in self.src_filter_list:
                       project_name = project.getAttribute('path').lstrip('/')
                       dependency_projects.add(project_name)
   
       return dependency_projects
   
   def getDependencyMap(self):
       return self.dependency_map