data engineers o paali itu randi

[mettastar]

meru vaade query engine (db) enti to serve users ?

high concurrency and low latency (sub second) unna db ethukuthunna..

any suggestions from your experience ?

 

TPS - 400 (400 requests per second support chesavi kaavali or close to that)

Latency - sub seconds to less than 5 secs okay

should be able to handle TBs of data 

SQL support unte manchidhi

[kasi]

kaka......asalu ni req enti, nuvu nosql db kosam looking aa???

do you know CAP theorem ???? if not check it out......

 

basic architecture iythea 

raw data ---> Spark/Hadoop(etl)--->nosql db ---> UX for users to generate reports  

 

[mettastar]

1 minute ago, kasi said:

kaka......asalu ni req enti, nuvu nosql db kosam looking aa???

do you know CAP theorem ???? if not check it out......

 

basic architecture iythea 

raw data ---> Spark/Hadoop(etl)--->nosql db ---> UX for users to generate reports  

 

Elasticsearch vaduthunnam kaani bulk loading chesthunte sasthundi adi. As it does indexing. 

Kylin ani okati undi explore chesa kaani adi olap cube based .. but ma data emo continues updates untayi historical data kuda .. rebuilding cubes is not easy adi kuda ruled out for now.

 

Any other nosql db in specific ? NoSqls tho problem endhante non key columns meda sorting or join btw other tables not easy run time lo

[kasi]

22 minutes ago, mettastar said:

Elasticsearch vaduthunnam kaani bulk loading chesthunte sasthundi adi. As it does indexing. 

Kylin ani okati undi explore chesa kaani adi olap cube based .. but ma data emo continues updates untayi historical data kuda .. rebuilding cubes is not easy adi kuda ruled out for now.

 

Any other nosql db in specific ? NoSqls tho problem endhante non key columns meda sorting or join btw other tables not easy run time lo

did you try increasing the resources.....scaling the server??? 

ES is distributed, so i think you can try increasing the sever resources 

[Spartan]

27 minutes ago, mettastar said:

Elasticsearch vaduthunnam kaani bulk loading chesthunte sasthundi adi. As it does indexing. 

Kylin ani okati undi explore chesa kaani adi olap cube based .. but ma data emo continues updates untayi historical data kuda .. rebuilding cubes is not easy adi kuda ruled out for now.

 

Any other nosql db in specific ? NoSqls tho problem endhante non key columns meda sorting or join btw other tables not easy run time lo

did u explore DocumentDB...

[mettastar]

11 minutes ago, Spartan said:

did u explore DocumentDB...

ninnane chadiva but use cases lo analytical ki vadachu ani mention cheyaledu .. bulk load ela untado chudali

[phoeniix]

5 minutes ago, mettastar said:

ninnane chadiva but use cases lo analytical ki vadachu ani mention cheyaledu .. bulk load ela untado chudali

elastic search enduku vadutunnaru....reports kosama kibana through generate cheyadam kosam

[phoeniix]

If not nosql you may consider Thoughspot BI tool but it is not an open source.

[phoeniix]

MariaDB too as a consumption layer .

[phoeniix]

57 minutes ago, kasi said:

kaka......asalu ni req enti, nuvu nosql db kosam looking aa???

do you know CAP theorem ???? if not check it out......

 

basic architecture iythea 

raw data ---> Spark/Hadoop(etl)--->nosql db ---> UX for users to generate reports  

 

typical architecture.

[Spartan]

20 minutes ago, mettastar said:

ninnane chadiva but use cases lo analytical ki vadachu ani mention cheyaledu .. bulk load ela untado chudali

https://segment.com/blog/choosing-a-database-for-analytics/

[Sarvapindi]

Data engineers more of development related..

[dewarist]

Apache Cassandra. A row partitioned distributed NO SQL data store.   We use it for storing inventory from thousands of locations and serve inventory lookup queries

What kind of data are you storing?

[mettastar]

31 minutes ago, dewarist said:

Apache Cassandra. A row partitioned distributed NO SQL data store.   We use it for storing inventory from thousands of locations and serve inventory lookup queries

What kind of data are you storing?

Sales invent forecast click stream customer behavior

[mettastar]

33 minutes ago, dewarist said:

Apache Cassandra. A row partitioned distributed NO SQL data store.   We use it for storing inventory from thousands of locations and serve inventory lookup queries

What kind of data are you storing?

Volume ey size lo untadi bro? And size of cluster? Query latency? Does it support ansi sql?