Data Quality

[csrcsr]

Just now, Konebhar6 said:

Entha confusion unte anta better kada jobs ki  Every new practice starts this way .. confusion ... Salim Pheku type projections ... FOMO by others ... We all saw what chat bots could do - nothing except frustrate users. They are getting better but still not there yet.  

In my opinion so far the best AI I saw was Digital Assistant ... Amazing notes by Microsoft Co-Pilot. We are very primitive right now ... but these technologies will emerge ... 

I work in pharma industry ... there are plenty of value addition AI/ML could do - Help business take decisions based on patterns found in data. 

coding lo matram its giving massive changes anna, maa dantlo co-pilot konni teams lo mandatory chesaru 

when we had stories most of the dev stories avg estimates used to be 50 % on actual dev , 20% on code coverage unit test and 20% on functional and integration tests  and rest 10% for documentation, product signoff etc 

with co pilot coming in some teams they were able to showcase acheive 100% code covarge  within no time and functyional integrations ttests they claim they were able to reduce 50% of the effort huge spike in developer communuty 

Google's code-assist from gemini is clamining to be more powerfuil in next year 

Yaa pharma medical lo already lot of simulations ata (i have no knowledge but telisina valu cheptunaru)

 

[Konebhar6]

3 minutes ago, Android_Halwa said:

Bro, do you know any such tech/companies at the conferences? 

We are looking at a few vendors for our data clean up but trust me, most vendors are just bluffing and we can see that in their presentations. Unfortunately, our data is so bad that we don't trust our own generated data and have been looking at alternate solutions for data cleanup tools or firms. Internally we tried our best but at best we can say its 80% clean. We are unable to leverage our data for the future at this stage...

You are right on vendors bluffing on capabilities. Informatica and Collibra has best tools so far on Data quality. I like Collibra better. 

We met with a company called AccelData but then again they also bluffed saying SAP plugins etc but they do not actually have it ready ... 

Alation, DQ Labs ... 

Like I said very primitive features. Informatica and Collibra can do basic anamoly detection based on standard deviation. You can add in rules to the data for any anomolies it finds. You can add your own rules based on your inputs from your stakeholders on the data sets. 

If you are on Snowflake, its cortex module has some built in features to do these. Probably MangoDB has those as well.

 

[Konebhar6]

12 minutes ago, Android_Halwa said:

Bro, do you know any such tech/companies at the conferences? 

We are looking at a few vendors for our data clean up but trust me, most vendors are just bluffing and we can see that in their presentations. Unfortunately, our data is so bad that we don't trust our own generated data and have been looking at alternate solutions for data cleanup tools or firms. Internally we tried our best but at best we can say its 80% clean. We are unable to leverage our data for the future at this stage...

Data lineage meeda chala tools unnai ... These tools have plugins to your ERP systems and core functional systems like SalesForce, HR systems, etc. They can build the structure and dependencies. They also have plugins to middleware systems/DBs/etc where the tool can analyze the code and build lineage automatically.

[lollilolli2020]

1 hour ago, Konebhar6 said:

My Take away from Informatica and SnowFlake conferences - Having good quality data is of utmost importance for AI and ML to succeed. Data Quality and Observability are the Industry terms being used to refer to this practice. Most companies will open positions related to this in coming years. Good space to enter for people with Analytical skills, DB Skills, Analyst etc.

Informatica and Collibra seem to be the front runners with decent platforms. A lot of new companies with niche features around this have come up. Have to see who will grow into a good tool/company.

ante naaku @Sucker ki good days ahead antava;; 

 

ee tool resume lo pettaloo cheppu; bega velli chatgpt ki prompt ista. 

[Konebhar6]

7 minutes ago, csrcsr said:

coding lo matram its giving massive changes anna, maa dantlo co-pilot konni teams lo mandatory chesaru 

when we had stories most of the dev stories avg estimates used to be 50 % on actual dev , 20% on code coverage unit test and 20% on functional and integration tests  and rest 10% for documentation, product signoff etc 

with co pilot coming in some teams they were able to showcase acheive 100% code covarge  within no time and functyional integrations ttests they claim they were able to reduce 50% of the effort huge spike in developer communuty 

Google's code-assist from gemini is clamining to be more powerfuil in next year 

Yaa pharma medical lo already lot of simulations ata (i have no knowledge but telisina valu cheptunaru)

 

Wow ... thats amazing .. 

I work in IT in a Pharma company. Anta cutting edge undadu. We rolled out co-pilot for business users and I heard it was not good. They were not happy and we ended up not buying those licenses. I did use it but mostly as digital assistant - notes taking and summarizing the meeting.

[Konebhar6]

Just now, lollilolli2020 said:

ante naaku @Sucker ki good days ahead antava;; 

 

ee tool resume lo pettaloo cheppu; bega velli chatgpt ki prompt ista. 

Collibra. I think ippudunna tools lo biggest name ide. You can also use Informatica Data Quality tool.

[lollilolli2020]

6 minutes ago, Konebhar6 said:

Collibra. I think ippudunna tools lo biggest name ide. You can also use Informatica Data Quality tool.

endii bro bra lu gatra antunnav ani @Sucker westindies nunchi antunnadu. 

good to know. 

mari adedo manastapam chendav DB ki ravatledu ani nee fans feel ayipoyaru inni rojulu. 

[vetrivel]

45 minutes ago, csrcsr said:

thats cool bro you are working on latest tech keep focussing on that, if possible when you have time can you have road map for learning ML AI tools for backend / front end develoeprs not like core math and scientist level but where we can conribute

Definitely brother 

 

I will definitely summarize how i started my journey

 

Ill be honest im a distributed systems engineer who had to do data science and ML 

 

 

 

 

[vetrivel]

45 minutes ago, Konebhar6 said:

Entha confusion unte anta better kada jobs ki  Every new practice starts this way .. confusion ... Salim Pheku type projections ... FOMO by others ... We all saw what chat bots could do - nothing except frustrate users. They are getting better but still not there yet.  

In my opinion so far the best AI I saw was Digital Assistant ... Amazing notes by Microsoft Co-Pilot. We are very primitive right now ... but these technologies will emerge ... 

I work in pharma industry ... there are plenty of value addition AI/ML could do - Help business take decisions based on patterns found in data. 

I always wondered how pharma companies used ML until i worked with Eli lily and Otsuka(Japan)

 

I didnt expect they would have so much data that they want to use 

[futureofandhra]

8 minutes ago, vetrivel said:

Definitely brother 

 

I will definitely summarize how i started my journey

 

Ill be honest im a distributed systems engineer who had to do data science and ML 

 

 

 

 

pm me also

[Android_Halwa]

41 minutes ago, Konebhar6 said:

Collibra. I think ippudunna tools lo biggest name ide. You can also use Informatica Data Quality tool.

We have heard about Collibra and its capacity. We did receive some good input from others about this vendor, thanks for the suggestion. 

[Konebhar6]

32 minutes ago, lollilolli2020 said:

endii bro bra lu gatra antunnav ani @Sucker westindies nunchi antunnadu. 

good to know. 

mari adedo manastapam chendav DB ki ravatledu ani nee fans feel ayipoyaru inni rojulu. 

I have learnt to ignore noise and have loads of patience. But sometimes over anipiste, I speak. Anthe.

Daniki, manasthapam anta scene endukulee ... Am in Bangkok now. Came for a vacation. "Inkenti mama full enjoy" eni anakandi .. All family here..

[lollilolli2020]

8 minutes ago, futureofandhra said:

pm me also

+1 @vetrivel

[Konebhar6]

1 minute ago, Android_Halwa said:

We have heard about Collibra and its capacity. We did receive some good input from others about this vendor, thanks for the suggestion. 

DQLabs ani oka vendor unnadu .. Talked to their architect. Very promising. Unless you get into their secret sauce, you dont really know if they have a good product. Check with them.

[Paamu]

1 hour ago, csrcsr said:

thats cool bro you are working on latest tech keep focussing on that, if possible when you have time can you have road map for learning ML AI tools for backend / front end develoeprs not like core math and scientist level but where we can conribute

Neeku reply oste … naakkuda pampu bro. Also em drink drinked ninna … full on fire.
Liquor shop ki poi bad liquor good liquor choose cheyyadam Antha easy kaadu bad data good data differ cheyyadam antunna @TuesdayStories