Get your ETL flow under Control using 3sigma limits.

Pavel Prudky (14.Apr.2018 at 13:30, 45 min)
Talk at Bulgaria Web Summit 2018 (English - UK)

Rating: 4 of 5

I would like to demonstrate how to use this simple statistical rule to capture outliers represented as invalid file loads in your etl process and pass them as documents into elasticsearch.setup will consist of Sql server hosting your etl Logging database, scheduled, stored procedure calculating The empirical rule and raising events into The windows event Log ,winlogbeat passing these events to elasticsearch and also kibana will be used for visualization. During The presentation, I would like to show how to get this setup running on your local machine to try out and talk further about The pros / cons of this solution.

Who are you?

Claim talk

Talk claims have been moved to the new Joind.in site.

Please login to the new site to claim your talk

Want to comment on this talk? Log in or create a new account or comment anonymously

Write a comment

 
Please note: you are not logged in and will be posting anonymously!
= seven plus nine

Comments

Rating: 4 of 5

18.Apr.2018 at 13:33 by Seatovic Dragan (8 comments) via Web2 LIVE

Whole system explained here is just an example (on windows platform) but similar approach is doable on linux and with various databases and monitoring systems. I like idea about statistical approach for alerting instead of fixed boundaries, so maybe presenter should have spend more time on useful statistic methods here then on winlogbeat and specific technologies.

© Joind.in 2018