Extracting SQL code from SSIS dtsx packages with Python lxml

#python #ssis #sql #lxml

Cross posted from my blog Analyze the Data not the Drivel

Lately, I’ve been refactoring a sprawling SSIS (SQL Server Integration Services) package that ineffectually wrestles with large XML files. In this programmer’s opinion using SSIS for heavy-duty XML parsing is geeky self-abuse so I’ve opted to replace an eye-ball straining¹ SSIS package with half a dozen, “as simple as possible but no simpler”, Python scripts. If the Python is fast enough for production great! If not the scripts will serve as a clear model² for something faster.

I’m only refactoring³ part of a larger ETL process so whatever I do it must mesh with the rest of the mess.

So where is the rest of the SSIS mess?

SSIS’s visual editor does a wonderful job of hiding the damn code!

This is a problem!

If only there was a simple way to troll through large sprawling SSIS spider-webby packages and extract the good bits. Fortunately, Python’s XML parsing tools can be easily applied to SSIS dtsx files. SSIS dtsx files are XML files. The following code snippets illustrate how to hack these files.

First import the required Python modules. lxml is not always included in Python distributions. Use the pip or conda tools to install this module.

# imports
import os
from lxml import etree

Set an output directory. I’m running on a Windows machine. If you’re on a Mac or Linux machine adjust the path.

# set sql output directory
sql_out = r"C:\temp\dtsxsql"
if not os.path.isdir(sql_out):
    os.makedirs(sql_out)

Point to the dtsx package you want to extract code from.

# dtsx files
dtsx_path = r'C:\Users\john\AnacondaProjects\testfolder\bixml'
ssis_dtsx = dtsx_path + r'\ParseXML.dtsx'

Read and parse the SSIS package.

tree = etree.parse(ssis_dtsx)
root = tree.getroot()

lxml renders XML namespace tags like <DTS:Executable as
www.microsoft.com/SqlServer/Dts\}Executable. The following gathers all the transformed element tags in the dtsx package.

# collect unique element tags in dtsx
ele_set = set()
for ele in root.xpath(".//*"):
    ele_set.add(ele.tag)    
print(ele_set)
print(len(ele_set))

Using transformed element tags of interest blast over the dtsx and suck out the bits of interest.

# extract sql code in source statements and write to *.sql files 
total_bytes = 0
package_name = root.attrib['{www.microsoft.com/SqlServer/Dts}ObjectName'].replace(" ","")
for cnt, ele in enumerate(root.xpath(".//*")):
  if ele.tag == "{www.microsoft.com/SqlServer/Dts}Executable":
    attr = ele.attrib
    for child0 in ele:
      if child0.tag == "{www.microsoft.com/SqlServer/Dts}ObjectData":
        for child1 in child0:
          sql_comment = attr["{www.microsoft.com/SqlServer/Dts}ObjectName"].strip()
          if child1.tag == "{www.microsoft.com/sqlserver/dts/tasks/sqltask}SqlTaskData":
            dtsx_sql = child1.attrib["{www.microsoft.com/sqlserver/dts/tasks/sqltask}SqlStatementSource"]
            dtsx_sql = "-- " + sql_comment + "\n" + dtsx_sql
            sql_file = sql_out + "\\" + package_name + str(cnt) + ".sql"
            total_bytes += len(dtsx_sql)
            print((len(dtsx_sql), sql_comment, sql_file))
            with open(sql_file, "w") as file:
              file.write(dtsx_sql)
print(('total sql code bytes',total_bytes))

The code snippets in this post are available in this Jupyter notebook: Extracting SQL code from SSIS dtsx packages with Python lxml. Download and tweak for your dtsx nightmare!

I frequently run into SSIS packages that cannot be viewed on 4K
monitors when fully zoomed out. ↩
Python’s readability is a major asset when disentangling
mess-ware. ↩
Yes, I’ve railed about the word “refactoring” in the past but I’ve moved on and so should you. “A foolish consistency is the hobgoblin of little minds.” ↩