Query to normalize table/combine row text
-
16-10-2019 - |
Pergunta
I have a table (call it oldTable) with columns like so:
ID (int),Rank (int),TextLineNumber (int),SomeText (varchar)
The primarykey is multi-part: ID+Rank+TextLineNumber.
I'm trying to transform/join it into another table (call it newTable) with columns like so:
ID (int), Rank (int), CombinedText (varchar)
and the primary key would be ID+Rank.
ID and Rank on the new table are already populated, but I need a query that would update the CombinedText column of the newTable with the following considerations:
- The Rank given on the new table may not exist on the old table, in which case it needs to pick the highest available rank from the old table that is not greater than the rank on the new table.
- The CombinedText column is a string concatenation of the "SomeText" column from the old table, concatenated in order of "TextLineNumber" using the Rank found from the first consideration.
Here's some example data:
old- http://i54.tinypic.com/jq0vmx.png
new- http://i53.tinypic.com/dhfyn8.png
I'm using MSSql 2005 if that matters. I currently do this using T-SQL and while loops, but it's become a serious performance bottle neck (taking about 1 minute for 10000 rows).
Edit: Expanded example data in CSV:
Old:
ID,Rank,LineNumber,SomeText
1,1,1,the qu
1,1,2,ick br
1,1,3,own
1,2,1,some te
1,2,2,xt
1,3,1,sample
2,7,1,jumped ov
2,7,2,er the
2,7,3,lazy
2,13,1,samp
2,13,2,le text
3,1,1,ABC
3,1,2,DEF
3,1,3,GHI
3,1,4,JKL
3,50,1,XYZ
New:
ID,Rank,CombinedText
1,2,some text
2,13,sample text
2,14,sample text
3,4,ABCDEFGHIJKL
3,5,ABCDEFGHIJKL
3,50,XYZ
3,55,XYZ
edit2:
Here's an example query that I found that does work but isn't fast enough (relying on multiple sub-queries):
update newtable set combinedtext =
coalesce ((select top 1 sometext from OldTable where OldTable.id=newtable.id and oldtable.rank=(select top 1 rank from oldtable where oldtable.id=newtable.id and oldtable.rank<=newtable.rank order by rank desc) and oldtable.linenumber=1),'') +
coalesce ((select top 1 sometext from OldTable where OldTable.id=newtable.id and oldtable.rank=(select top 1 rank from oldtable where oldtable.id=newtable.id and oldtable.rank<=newtable.rank order by rank desc) and oldtable.linenumber=2),'') +
coalesce ((select top 1 sometext from OldTable where OldTable.id=newtable.id and oldtable.rank=(select top 1 rank from oldtable where oldtable.id=newtable.id and oldtable.rank<=newtable.rank order by rank desc) and oldtable.linenumber=3),'') +
coalesce ((select top 1 sometext from OldTable where OldTable.id=newtable.id and oldtable.rank=(select top 1 rank from oldtable where oldtable.id=newtable.id and oldtable.rank<=newtable.rank order by rank desc) and oldtable.linenumber=4),'') +
coalesce ((select top 1 sometext from OldTable where OldTable.id=newtable.id and oldtable.rank=(select top 1 rank from oldtable where oldtable.id=newtable.id and oldtable.rank<=newtable.rank order by rank desc) and oldtable.linenumber=5),'')
It also assumes a max line number of 5 which may not be the case. I don't mind hard-coding the linenumbers in all the way to a max of 20 if that's what it takes, but ideally it would be able to account for them differently. Getting execution time under 20 seconds (the actual data) is the goal...
Solução
This should work, I will clean it up later so its more efficient.
DECLARE @Old TABLE (
id INT,
rank INT,
linenumber INT,
sometext VARCHAR(1000))
DECLARE @New TABLE (
id INT,
rank INT,
combinedtext VARCHAR(1000))
;WITH combinedresults(ctid, id, rank, linenumber, combinedtext)
AS (SELECT 0,
id,
rank,
linenumber,
CAST (sometext AS VARCHAR(8000))
FROM @Old o
WHERE NOT EXISTS (SELECT TOP 1 1
FROM @Old
WHERE id = o.id
AND rank = o.rank
AND linenumber < o.linenumber)
UNION ALL
SELECT ctid + 1,
o.id,
o.rank,
o.linenumber,
ct.combinedtext + o.sometext
FROM @Old o
INNER JOIN combinedresults ct
ON ct.id = o.id
AND ct.rank = o.rank
WHERE o.linenumber > ct.linenumber)
UPDATE n
SET combinedtext = ct.combinedtext
FROM @New n
INNER JOIN (SELECT n.id,
n.rank,
MAX(o.rank) orank
FROM @new n
INNER JOIN @Old o
ON n.id = o.id
AND o.rank <= n.rank
GROUP BY n.id,
n.rank) r
ON n.id = r.id
AND n.rank = r.rank
INNER JOIN (SELECT id,
ct.rank,
MAX(ctid) ctid
FROM combinedresults ct
GROUP BY ct.id,
ct.rank) r2
ON r2.id = r.id
AND r2.rank = r.orank
INNER JOIN combinedresults ct
ON r.id = ct.id
AND ct.rank = r.orank
AND ct.ctid = r2.ctid
SELECT *
FROM @New
Outras dicas
You can create a function that stings the values together using a cursor within the function, but that's about your only option. You'll need to do row by row processing to make this happen.
I'm not good with CTE yet, so here's my take on the question using a more traditional approach without cursors.
Requirement #2 reminds me of a project I worked on that required producing a comma-separated concatenation of the values on a column grouped by some category. The solution I used required a UDF to produce the concatenated string using the provided category id.
Below is the adapted UDF using ID and Rank parameters:
CREATE FUNCTION fnCombinedText
(
@ID int,
@Rank int
)
RETURNS varchar(MAX)
AS
BEGIN
DECLARE @CombinedText varchar(MAX)
SET @CombinedText = ''
SELECT @CombinedText = @CombinedText + SomeText
FROM oldTable
WHERE [ID] = @ID
AND [Rank] = @Rank
ORDER BY [Rank]
RETURN @CombinedText
END
Requirement #1 can be accomplished by joing the newTable Rank's with all distinct equal or less oldTable Rank's and getting the best/top match:
CREATE TABLE #RankMap
(
newID int,
newRank int,
oldID int,
oldRank int
)
INSERT INTO #RankMap
SELECT newID, newRank, oldID, oldRank
FROM
(
SELECT
n.id AS newID,
n.rank AS newRank,
o.id AS oldID,
o.rank as oldRank,
RANK() OVER(PARTITION BY n.rank ORDER BY o.rank DESC) AS topRank
FROM
newtable AS n
LEFT OUTER JOIN (SELECT DISTINCT id, rank FROM oldtable) AS o
ON n.id = o.id
AND n.rank >= o.rank
) AS matchEqualLess
WHERE topRank = 1
Now that we have the mapped oldTable Rank's, we can use the UDF to generate the CombinedText's:
SELECT
newID,
newRank,
dbo.fnCombinedText(oldID, oldRank) AS CombinedText
FROM #RankMap
--Below is the resultset:
newID newRank CombinedText
----------- ----------- --------------------
1 2 some text
3 4 ABCDEFGHIJKL
3 5 ABCDEFGHIJKL
2 13 sample text
2 14 sample text
3 50 XYZ
3 55 XYZ
The main downside for this solution is that each call to the UDF fnCombinedText() is essentially a separate SELECT on the oldTable. I bet this approach can be ported to a more scalable CTE query. And I guess I should really get around to mastering CTE, too.