As a powershell newbie I’m pretty happy with this one, even if it relies on calling the split utility from cygwin and a dos copy command to perform… The split.exe is the windows port of the unix split command, I downloaded this as part of cygwin. However I didn’t want to install cygwin on the server so just copied the split.exe file across. This performs pretty well on our VM, it processed 200GB of text files into 64MB chunks in about 2 hours. It’s a one off process to seed the historical invoices into the DWH, so I have need to seek further enhancements.
Recent Posts
Capturing load errors with SSIS / SQL2016
Using SQL 2016 and SSDT for Visual Studio 2015 I’ve been looking for a few weeks how to get an error in loading a .csv file to show the offending column, and the name of the error. There were some improvements to make this possible via a script object in SQL 2016, however they’ve moved a few of the functions around which made the code posted online, including on the MS site, outdated. Here’s the C# code that is working with the latest releases:
SSIS – Checking if a file exists
I’ve found that SSIS will throw an exception if the CSV file I want to load is missing. While I do have a script that checks all my connections, this doesn’t seem to do enough for text file data sources and lets them pass. The solution I’ve found is using Powershell to test for the file and return the result to SSIS.
Another SSIS Framework – Beginner’s Guide
Another SSIS Framework (for n00bs) While there is some good high level documentation written by SqlArchitect, I hope for this guide to compliment his efforts by providing some step by step step instructions for those new to SSIS against a clean SQL 2016 environment using SSISDB.
SSIS Framework Selection
After 16 years of BI development in the IBM Cognos stack, I’ve recently jumped ship over to Microsoft and have faced a pretty steep learning curve as I plan to build a robust data warehouse and reporting solution in three months! To jump start this activity I’ve been reviewing SSIS Frameworks, which will provide the ‘belts and braces’ around ETL code without my having to invest months building this logic around logging, notification, error detection / recovery and job flow from scratch.
Bursting Gotcha
After spending what seems like ages trying to troubleshoot why an account does not have access to burst a report, I thought I’d share the solution. In addition to setting up the capability for a user to burst content, this is also a capability at the package level. Duh.
Event Viewer Filters Part 2
Still playing with the Event Viewer, I’ve written a bunch of custom filters that give a great insight into how Cognos is behaving, especially around monthly patching. I’ve saved these queries on my local machine and use them against all our servers by right-clicking on the event viewer root and ‘Connect to another computer’. Cognos Service This query shows activity of the ‘IBM Cognos’ service Server Restarts USER32 events appear to show all restarts against the server Windows Updates (Patches) This event shows me all Patches being applied to the server
Cognos Services Manager
Here’s a batch file I’ve written over the past few years that automates the starting, stopping, testing and restarting of distributed Cognos services across three environments. This particular script has been anonymised, but is hopefully still in working order. It’s been in use for ~3 years on Windows 2008 R2 Server.
TM1 Server Crash Notification via Event Viewer Scheduled Tasks
I’ve just put together an XML query that filters the event viewer to just show TM1 application crashes. This can be used as a custom view in Event Viewer to see all TM1 application crashes, and it can be into a scheduled task to take other action, such as sending an email to interested developers, or maybe even restarting the service.
Find who’s been tampering in the content store
Today I was poking around in Production and noticed the modified date on a report of yesterday. After asking around my team, it became apparent that none of us had made the change… was there a security hole? This query allowed me to identify who made the change… and as it turns out the modified date can be updated by quite a few scenarios in Cognos, including any user creating a view in their My Folders. Oh well, here’s the query for next time!
Twitter
G+
LinkedIn
Recent Comments