Post Tagged with: “powershell”

Script to split huge text file into chunks with headers

Script to split huge text file into chunks with headers

As a powershell newbie I’m pretty happy with this one, even if it relies on calling the split utility from cygwin and a dos copy command to perform… The split.exe is the windows port of the unix split command, I downloaded this as part of cygwin. However I didn’t want to install cygwin on the server so just copied the split.exe file across. This performs pretty well on our VM, it processed 200GB of text files into 64MB chunks in about 2 hours. It’s a one off process to seed the historical invoices into the DWH, so I have┬áneed to seek further enhancements.

read more

SSIS – Checking if a file exists

SSIS – Checking if a file exists

I’ve found that SSIS will throw an exception if the CSV file I want to load is missing. While I do have a script that checks all my connections, this doesn’t seem to do enough for text file data sources and lets them pass. The solution I’ve found is using Powershell to test for the file and return the result to SSIS.

read more