How is testing *system test and uat done in big data environments

#1

ARe there separate test clusters .i.e separate name node… data nodes… how is spark and scala testing done… ? or lets say there is a map reduce code… how will test team test it ? I am just thinking out loud here…as I have not worked actually in big data space

0 Likes

#2

On similar lines other question is how is debugging on a scala script done ?

0 Likes

#3

There are no sophisticated tools for unit testing for Data processing.

ARe there separate test clusters .i.e separate name node… data nodes… how is spark and scala testing done… ? or lets say there is a map reduce code… how will test team test it ?

1st method: Have separate clusters for development, testing and production where each environment have its own set of nodes such as namenode, datanode etc.

2nd method: Multi tenancy - By using capacity or fair scheduler one can dedicate lower percentage of resources for testing

Each of it have its own advantage and disadvantages. But both are common methods.

0 Likes

#4

There will be separate QA server(clusters) instance to perform testing process.

0 Likes

#5

What about MRunit tool? They are using in the industry or just available in the market.

0 Likes