ããŒã¿ã®æ±ããããžãã¹ã«ãããå¿ é ã¹ãã«ãšãªããããŒã¿ãšã³ãžãã¢ã®éèŠã¯ãããŸã§ä»¥äžã«é«ãŸã£ãŠããŸããããããå€ãã®ããžãã¹ããŒãœã³ã«ãšã£ãŠãããŒã¿ãšã³ãžãã¢ãšããŠã®ç¢ºåºãããã£ãªã¢ãã¹ã圢æããã«ã¯ãã©ã®ãããªã¹ãã«ãå¿ èŠãªã®ã§ããããïŒ ãã®èšäºã§ã¯ãããŒã¿ãšã³ãžãã¢ãšããŠã®åžå ŽäŸ¡å€ã®ç解ããå§ãŸããå¿ èŠãªæè¡ãšãœããã¹ãã«ãå¹æçãªè»¢è·æŠç¥ãŸã§ã転è·ãæåã«å°ããã€ã³ããæãããããã€ç°¡æœã«ãŸãšããŠãããŸããããªãã®æ¬¡ã®ãã£ãªã¢ã¹ãããããããæŠç¥çã«èšç»ããŸãããã
ããŒã¿ãšã³ãžãã¢ãšã¯ïŒãã£ãªã¢ã®é åãšåžå ŽäŸ¡å€
ããŒã¿ãšã³ãžãã¢ãšããè·æ¥ã泚ç®ãããŠããçŸä»£ãå€ãã®æ¥çã§ããŒã¿ã®äŸ¡å€ãé«ãŸã£ãŠããŸããå°éçãªã¹ãã«ã»ãããæã€ããŒã¿ãšã³ãžãã¢ã¯ãäŒæ¥ã«ãšã£ãŠäŸ¡å€ããè³ç£ãšãªã£ãŠããããã®ãã£ãªã¢ãã¹ã¯å€å²ã«ããããŸãã
ããŒã¿ãšã³ãžãã¢ã¯ãçããŒã¿ãåéãä¿åãåŠçããåæè ãããžãã¹ãŠãŒã¶ãŒã容æã«ã¢ã¯ã»ã¹ã§ãã圢ã«æŽããå°é家ã§ãã圌ãã¯ããŒã¿ãŠã§ã¢ããŠã¹ã®æ§ç¯ãETLããã»ã¹ã®ç®¡çãããã°ããŒã¿æè¡ã®é©çšãªã©ãããŒã¿ã«é¢ããå¹ åºãæ¥åãæ ããŸãã
åžå ŽäŸ¡å€ãšããåŽé¢ããèŠããšãããŒã¿ãšã³ãžãã¢ã¯é«ãå°éæ§ãèæ¯ã«ãäŒæ¥ã®ææ決å®ãæ¯ããéèŠãªåœ¹å²ãæãããŠããŸããããããèæ¯ããããŒã¿ãšã³ãžãã¢ã¯é«ã絊äžãåŸãããšãã§ãããŸãããã£ãªã¢å±æãåºãã£ãŠããã®ã§ãã
ããŒã¿ãšã³ãžãã¢ãªã³ã°ã®åºç€ç¥è
ããŒã¿ãšã³ãžãã¢ãªã³ã°ã®åºç€ç¥èãšã¯ãããŒã¿ããŒã¹ç®¡çãããã°ã©ãã³ã°ãã·ã¹ãã ã€ã³ãã©ãããŒã¿ã¢ããªã³ã°ãªã©ãããŒã¿ãæ±ãããã®åºæ¬ãšãªãã¹ãã«çŸ€ã®ããšã§ãããããã®ã¹ãã«ã¯ãããŒã¿ãšã³ãžãã¢ãšããŠã®åºç€ãç¯ãäžããããã«äžå¯æ¬ ã§ãã
ãã®ä»ã«ããã¯ã©ãŠãã³ã³ãã¥ãŒãã£ã³ã°ãµãŒãã¹ã®ç¥èããããã°ããŒã¿ãã¯ãããžãŒã«é¢ããç解ããããŒã¿ãšã³ãžãã¢ã«ãšã£ãŠéèŠãªã¹ãã«ã»ããã§ããHadoopãSparkãªã©ã®æè¡ã¯ãç¹ã«æ±ããããã±ãŒã¹ãå€ãèŠãããŸãã
åºç€ç¥èãç¿åŸããã«ã¯ããªã³ã©ã€ã³ã³ãŒã¹ãå°éã®ãã¬ãŒãã³ã°ããã°ã©ã ãæå¹ã§ãããããã®ããã°ã©ã ãéããŠãç¶ç¶çã«åŠç¿ãè¡ãããšã§ãæšä»ã®æè¡é²åã«è¿œãã€ããåžå ŽäŸ¡å€ãåäžãããããšãã§ããŸãã
ãã£ãªã¢ãã¹ãšé²ãã¹ãé
ããŒã¿ãšã³ãžãã¢ã®ãã£ãªã¢ãã¹ã¯å€æ§ã§ãçµéšãã¹ãã«æ¬¡ç¬¬ã§ããã€ãã®é²è·¯ããããŸããåçŽè ã¯ããŒã¿ã¢ããªã¹ãããžã¥ãã¢ããŒã¿ãšã³ãžãã¢ããã¹ã¿ãŒãããåŸã ã«ã·ãã¢ããŒã¿ãšã³ãžãã¢ãããŒã¿ãµã€ãšã³ãã£ã¹ããžãšã¹ãããã¢ããããããšãå¯èœã§ãã
äžã«ã¯ããŒã ãªãŒããŒãããŒã¿ãšã³ãžãã¢ãªã³ã°ãããŒãžã£ãŒãžãšãã£ãªã¢ãé²ããè ãããŸãããããã®ããžã·ã§ã³ã§ã¯ãããžã§ã¯ã管çãããŒã éå¶ã®ã¹ãã«ãæ±ããããããæŠç¥çãªèŠç¹ã§ããŒã¿ãœãªã¥ãŒã·ã§ã³ãæäŸããèœåãå¿ èŠã«ãªããŸãã
ãŸããå°éæ§ãæ·±ããããšã§ã€ã³ããã³ãã³ãã³ã³ãµã«ã¿ã³ããšããŠç¬ç«ããè€æ°ã®äŒæ¥ã«å¯ŸããããŒã¿ãœãªã¥ãŒã·ã§ã³ãæäŸããéãéããŸãããã£ãªã¢ã®æ¹åæ§ã¯åã ã®ç®æšãèå³ã«å¿ããŠå¹ åºãéžã¶ããšãã§ããã®ã§ãã
éèŠå¢å ã«äŒŽãåžå ŽäŸ¡å€ã®é«ãŸã
ããžã¿ã«ãã©ã³ã¹ãã©ãŒã¡ãŒã·ã§ã³ãé²è¡ããäžã§ãããŒã¿ãé©åã«ç®¡çã»æŽ»çšããããšãäŒæ¥ã®ç«¶äºåãå·Šå³ããããã«ãªããŸããããã®ååã¯ãããŒã¿ãšã³ãžãã¢ã®éèŠå¢å ã«çŽçµããåžå ŽäŸ¡å€ã®é«ãŸãããããããŠããŸãã
åç£æ¥åéã§ã¯ããŒã¿ã®éãççºçã«å¢å ããŠãããããããµããŒãããããŒã¿ã€ã³ãã©ãã¢ããªãã£ã¯ã¹ã®ããŒãºãåæ§ã«å¢ããŠããŸããããŒã¿ãšã³ãžãã¢ã¯ããããã®ããŒãºã«å¯Ÿå¿ããããã«äžå¯æ¬ ãªåœ¹å²ãæãããŠããã®ã§ãã
å ããŠãæ°ããªããŒã¿ãœãŒã¹ã®æ¢çŽ¢ãæ©æ¢°åŠç¿ã®é©çšãªã©ãããŒã¿ãšã³ãžãã¢ã®åœ¹å²ã¯æ¡å€§ãç¶ããŠãããã¹ãã«ãæã¡ç¶ããããšã§çµŠäžã®åäžãããé«åºŠãªè·åãžã®ãã£ã³ã¹ãå¢ããŠããŸãã
ããŒã¿ãšã³ãžãã¢ã®ãã£ãªã¢ã§åŸãããé å
ããŒã¿ãšã³ãžãã¢ãšããŠã®ãã£ãªã¢ã¯ãå°éçãªæè¡ã磚ãã°ç£šãã»ã©ããã®ã¹ãã«ãéå®ãããåéã§ããããšãããè·æ¥ãšããŠã®é åãéåžžã«é«ãã§ããã¹ãã«ã®å°éæ§ãèªåã®åžå ŽäŸ¡å€ãçŽæ¥çã«äžããèŠå ãšãªããããã¹ãã«åäžãžã®ã¢ãããŒã·ã§ã³ãæç¶ããããã§ãã
ããã«ãããŒã¿ãæ±ãããšã®ã§ããäŒæ¥ã¯å¹ åºãæ¥çã«æž¡ã£ãŠãããéžã¹ãè·å Žã®å€æ§æ§ãããŒã¿ãšã³ãžãã¢ã®ãã£ãªã¢ã«è±ãããå ããŠããŸããäŸããŠèšããªãã°ãéèããã«ã¹ã±ã¢ããã¯ãããžãŒãªã©ãã»ãŒå šãŠã®ã»ã¯ã¿ãŒã§åœŒãã®ã¹ãã«ã¯å¿ èŠãšãããŠããŸãã
è·åã®èªç±åºŠãé«ãããšãé åã®äžã€ã§ãããã«ã¿ã€ã ãå¥çŽããªã¢ãŒãã¯ãŒã¯ãªã©ãåãæ¹ãèªåã®ã©ã€ãã¹ã¿ã€ã«ã«åãããŠèª¿æŽã§ããæè»æ§ããå€ãã®ããŒã¿ãšã³ãžãã¢ã«ãšã£ãŠéèŠãªã¡ãªãããšãªã£ãŠããŸãã
転è·åã«æŽçãã¹ãããŒã¿ãšã³ãžãã¢ã®ã¹ãã«ã»ãã
å¿ é ãšãªãæè¡ã¹ãã«
ããŒã¿ãšã³ãžãã¢ãšããŠè»¢è·ãæåãããããã«ã¯ãããŒã¿ããŒã¹ç®¡çãETLïŒExtractãTransformãLoadïŒããã»ã¹ã®ç解ãåã³ããŒã¿ãŠã§ã¢ããŠãžã³ã°ã«é¢ããå å®ãªç¥èãå¿ èŠäžå¯æ¬ ã§ãããããã¯ããŒã¿ã®åãæ±ãã«çŽæ¥é¢ä¿ããŠãããã©ã®æ¥çã§ãã£ãŠãæ±ããããã¹ãã«ã§ãã
ãŸãåæããµããŒãããããã®ããŒã¿ã¢ããªã³ã°ã®æèœãéèŠã§ããããŒã¿ãæŽçããé©åãªãã©ãŒãããã«å€æããåã¯ãäŒæ¥ãçŽé¢ããåé¡ã解決ããããã§åºç€ãšãªããŸããããã«ãã¯ã©ãŠããµãŒãã¹ãžã®ç解ãæ·±ããããšããçŸä»£ã®ããŒã¿ãšã³ãžãã¢ãªã³ã°ã®äžçã§ã¯æ±ããããã¹ãã«ã®äžã€ã§ãã
ããŒã¿ã»ãã¥ãªãã£ã«é¢ããç¥èãéåžžã«éèŠã§ããããŒã¿ã®ä¿è·ãšãã©ã€ãã·ãŒã¯ã©ã®äŒç€Ÿã«ãšã£ãŠãæåªå äºé ãšãªã£ãŠãããæè¡çãªå¯Ÿçãç解ã»å®è£ ã§ããèœåãå¿ èŠãšãããŸãã
ãœããã¹ãã«ãšãã®éèŠæ§
æè¡ã¹ãã«ã ãã§ã¯ãªããã³ãã¥ãã±ãŒã·ã§ã³èœåããŸãããŒã¿ãšã³ãžãã¢ã«ãšã£ãŠå¿ èŠãªè³è³ªã§ããããŒã å ã®éçºè ãããžãã¹ãµã€ãã®äººéãšå¹çããååããåã¯ããããžã§ã¯ããæåã«å°ãã«ã®ãšãªããŸãã
åé¡è§£æ±ºèœåããŸããããŒã¿ãšã³ãžãã¢ãªã³ã°ã«ãããŠäžå¯æ¬ ãªãœããã¹ãã«ã§ããããŒã¿ã«ã¯äºæž¬äžèœãªåé¡ãã€ããã®ã§ãããåµé çã§æè»ãªæèãæ±ããããå Žé¢ãå€ã ãããŸãã
ããã«ãããžã§ã¯ã管çèœåããããŒã¿ãšã³ãžãã¢ãå¹ççã«äœæ¥ãé²ããããã§åœ¹ç«ã€ã¹ãã«ã§ããè€æ°ã®ã¿ã¹ã¯ãåæã«å¶åŸ¡ããåªå é äœãã€ããæéå ã«ææãåºãèœåãæ±ããããŸãã
åŠã¶ã¹ãããã°ã©ãã³ã°èšèªãšããŒã«
SQLã¯ããŒã¿æäœã®åºç€ã§ãããããç¿åŸã¯å¿ é ã§ãã倧éã®ããŒã¿ã»ããã§äœæ¥ããéã«ã¯ãPythonãJavaã®ãããªããã°ã©ãã³ã°èšèªãæå¹ã§ãããããã¯ããŒã¿åŠçãšåæã¿ã¹ã¯ãèªååããŸãã
ããŒã¿ãšã³ãžãã¢ãªã³ã°ã®ãã£ãŒã«ãã§ãã䜿ãããããŒã«ã«ã¯Apache HadoopãApache SparkããããŸãããããã®å€§èŠæš¡ããŒã¿åŠçãã¬ãŒã ã¯ãŒã¯ã¯ãæ¥åã®çç£æ§ãå€§å¹ ã«é«ããããšãã§ããŸãã
ãŸããããŒã¿ãã€ãã©ã€ã³ã®èªååã«ã¯ãApache AirflowãLuigiã®ãããªã¯ãŒã¯ãããŒç®¡çããŒã«ã®ç解ã圹ã«ç«ã¡ãŸãããããã®ã¹ãã«ã¯ãå¹ççãªããŒã¿ãããŒã®æ§ç¯ã«çŽæ¥çãªäŸ¡å€ããããããŸãã
å®åçµéšãšãã®ç©ã¿æ¹
çè«çãªç¥èã ãã§ãªããå®éã®æ¥åã§ã®çµéšãããŒã¿ãšã³ãžãã¢ã®è»¢è·ã«ã¯å€§ããªåœ±é¿ãäžããŸããæ°ããããžã·ã§ã³ã§æ±ããããå®åçµéšãç©ãããã«ã¯ãã€ã³ã¿ãŒã³ã·ãããããªãŒã©ã³ã¹ã®ãããžã§ã¯ããæãã¯ãªãŒãã³ãœãŒã¹ãããžã§ã¯ããžã®åå ãæå¹ã§ãã
ããŒããã©ãªãªãæ§ç¯ããããšã«ãã£ãŠãçµéšã蚌æããæœåšçãªéçšäž»ã«èªåã®èœåãã¢ããŒã«ã§ããŸããå ·äœçãªãããžã§ã¯ããå®éãããã®ããã»ã¹ãšçµæã説æã§ããããšãéèŠã§ãã
ãŸããã³ãã¥ããã£ã€ãã³ããããŒãã¢ããã«åå ããããšã§ãçŸå Žã§ã®çµéšãããã«æ·±ããããšãã§ããŸãããããã¯åŠã³ãšåæã«ãæ¥çã®ãããã¯ãŒãã³ã°ã«ãå©çšã§ãã貎éãªæ©äŒã§ãã
å¹æçãªããŒã¿ãšã³ãžãã¢ã®è»¢è·æŠç¥
åžå Žãšããžã·ã§ã³ã®ãªãµãŒãæ¹æ³
ããŒã¿ãšã³ãžãã¢ãšããŠæåãã転è·ãæããããã«ã¯ããŸãçŸåšã®åžå Žç¶æ³ãææ¡ããããšãéèŠã§ããæ±äººãµã€ããæ¥çã®ãã¥ãŒã¹ãè¿œããã©ã®äŒæ¥ãããŒã¿ãšã³ãžãã¢ãæ±ããŠãããããŸããã®äŒæ¥ã®ããžãã¹ã¢ãã«ãæåãç 究ããŸãããã
次ã«ãæ±ããããã¹ãã«ã»ãããçµéšãåæããŸããæ§ã ãªæ¥çã§ããŒã¿ãšã³ãžãã¢ã®ããŒãºã¯ç°ãªããããç¹å®ã®ããŒã¿ããŒã¹ç®¡çã·ã¹ãã ãããã°ã©ãã³ã°èšèªãããŒã¿åŠçãã¬ãŒã ã¯ãŒã¯ã«é¢é£ããæè¡ãææ¡ããããšã倧åã§ãã
ãŸãã¯ããããã¯ãŒãã³ã°ã掻çšããŠæ¥çã®å æ ãæªå ¬éæ±äººãæ¢ãããšã1ã€ã®æã§ããåæ¥ä»ç€Ÿã®ããŒã¿ãšã³ãžãã¢ããã£ãªã¢ã€ãã³ããªã©ãéããŠæ å ±ãéããåžå Žã®é絊ãã©ã³ã¹ãç解ããããšãæŠç¥çãªè»¢è·ã«ç¹ãããŸãã
å±¥æŽæžã»è·åçµæŽæžã®æžãæ¹ã®ãã€ã³ã
å±¥æŽæžãè·åçµæŽæžãäœæããéã¯ãæ確ã§ç°¡æœãªè¡šçŸãå¿æããããšãéèŠã§ããæå¹ãªãããžã§ã¯ã管çææ³ãããŒã¿ããŒã¹ç®¡çãªã©ãå ·äœçãªã¹ãã«ãææã瀺ãããŒã¯ãŒããåé¢ã«åºããŸãããã
ãŸããã©ã®ããã«ããŠããžãã¹ã«äŸ¡å€ããããããŠãããã«çŠç¹ãåœãŠãŠãã ãããããŒã ã§ã®æ¥åéè¡ããåé¡è§£æ±ºèœåãå ·äœçãªäŸãçšããŠèšè¿°ããããšã§ãèªèº«ã®èœåãã¢ããŒã«ããããšãã§ããŸãã
çµéšè±ããªããŒã¿ãšã³ãžãã¢ã§ããã°éå»ã®ãããžã§ã¯ãã®èŠæš¡ã圱é¿ãæ°æè¡ã®å°å ¥çµéšãªã©ãæ°å€ãäºäŸãçšããŠéææãäŒããããšãæšå¥šãããŸããèªãã®æ¥çžŸã客芳çã«è©äŸ¡ãããããå±¥æŽæžãè·åçµæŽæžã«åæ ãããããšãã«ã®ãšãªããŸãã
é¢æ¥ã§ã¢ããŒã«ãã¹ããã€ã³ã
é¢æ¥ã§ã¯ãæè¡çãªã¹ãã«ã¯ãã¡ããã§ãããã³ãã¥ãã±ãŒã·ã§ã³èœåãããŒã ã§ã®ååã¹ãã«ãã¢ããŒã«ããããšã倧åã§ããå®åã§è€éãªããŒã¿åé¡ã解決ããçµéšããããŒã å ã§ã®ãªãŒããŒã·ãããçºæ®ããäŸãªã©ãæºåããŠãããŸãããã
ããã«ãé¢æ¥å®ãäŒæ¥ã®ããŒãºã課é¡ãç解ããŠãããè©ŠãããããšããããŸãããªãµãŒããéããŠç¥ãåŸãäŒæ¥ã®æ å ±ãããšã«ãèªããã©ã®ããã«äŸ¡å€ãæäŸã§ããããè¿°ã¹ããšå¹æçã§ãã
æåŸã«ãå°æ¥æ§ããã£ãªã¢ãã©ã³ã®èŠ³ç¹ãããèªèº«ã®é·æããžã§ã³ãèªãããšãããªãã®ç±æãäŒæ¥ãžã®ã³ãããã¡ã³ããæããããããšã§ãé¢æ¥å®ã«è¯ãå°è±¡ãæ®ãããšãã§ããã§ãããã
転è·æŽ»åã§å¿ãããã¹ããã€ã³ãã»ãã
転è·æŽ»åã¯ãã°ãã°ã¹ãã¬ã¹ã䌎ãããšããããŸããããããããžãã£ããªå§¿å¢ãä¿ã¡ã倱æãæããã«åæ©äŒã«æãããšã倧åã§ããæåãèŠããŠãããŸã§ã¯å¿èãå¿ èŠã«ãªãå Žé¢ãå€ã ãããŸãã
ãŸãã転è·ã¯ãã ã®ä»äºã®å€æŽã§ã¯ãªãããã£ãªã¢ãã¹ã圢äœãéèŠãªã¹ãããã§ããèªå·±åæãè¡ããæ¬åœã«æ±ãããã£ãªã¢ãç掻ã®è³ªãèŠæ¥µããããšãæåãžã®éµã§ãã
çµããã«ã倱æãæããããªã¹ã¯ãåãããšãæé·ãžã®éãšãªããŸããäžç¢ºå®ãªç¶æ³ã«ãæè»ã«å¯Ÿå¿ããæ°ããªææŠã楜ãããã€ã³ãã»ããããæçµçã«ã¯å ±ãããããšãä¿¡ããŠèšã¿ãŸãããã
ããŒã¿ãšã³ãžãã¢ãç®æãã¹ãæ¥çãšäŒæ¥
泚ç®ã®æ¥çãšãã®ç¹åŸŽ
ããŒã¿ãšã³ãžãã¢ã«ãšã£ãŠæ³šç®ãã¹ãæ¥çã¯ããã¯ãããžãŒãéèããã«ã¹ã±ã¢ãæããããŸãããã¯ãããžãŒæ¥çã§ã¯ãå»ã ãšå€åããåžå Žã®ããŒãºã«å¯Ÿå¿ããããã«ããŒã¿ã®æŽ»çšãäžå¯æ¬ ã§ããéèæ¥çã§ã¯ããªã¹ã¯ç®¡çã顧客è¡åã®åæãéããŠè£œåéçºãè¡ã£ãŠãããããŒã¿ãšã³ãžãã¢ã®åœ¹å²ãéèŠã«ãªã£ãŠããŸãããŸããã«ã¹ã±ã¢æ¥çã§ã¯ãæ£è ã®æ²»çå±¥æŽãèšåºè©Šéšã®çµæãªã©èšå€§ãªéã®ããŒã¿ãåŠçããå»çãµãŒãã¹ã®è³ªã®åäžãæ±ããããŠããŸãã
ãããã®æ¥çã§ã¯ãããŒã¿ãšã³ãžãã¢ãããã°ããŒã¿ãæ±ãããã«å¿ èŠãªæè¡åãšãšãã«ãæ¥çç¹æã®ç¥èããå¿ èŠãšãããŸããåŸã£ãŠãããããã®ããŒã¿ãšã³ãžãã¢ã¯å°éæ§ãé«ããªãããæ¥çååã«ã粟éããå¿ èŠãããã§ãããã
ããã«ãããã®ç£æ¥ã«å ±éããç¹åŸŽã¯ãããŒã¿ããªãã³ãªææ決å®ãè¡ãæåã確ç«ãããŠããç¹ã§ããããã«ãããããŒã¿ãšã³ãžãã¢ã¯ããæŠç¥çãªåœ¹å²ãæ ããäºæ¥æé·ã®åååãšããŠæåŸ ãããŠããŸãã
æ±ããããããŒã¿ãšã³ãžãã¢ã®çš®é¡
ããŒã¿ãšã³ãžãã¢ã«ã¯ãç°ãªãã¹ãã«ã»ãããå°éç¥èãèŠæ±ãããããã€ãã®çš®é¡ããããŸããäžè¬çãªããŒã¿ãšã³ãžãã¢ã¯ããŒã¿ããŒã¹ãããŒã¿ãŠã§ã¢ããŠã¹ã®ç®¡çã«é¢é£ããæè¡ã«ç²ŸéããŠããå¿ èŠããããŸãããŸããããŒã¿ãã€ãã©ã€ã³ã®æ§ç¯ãããŒã¿ã®ETLïŒæœåºãå€æãããŒãïŒåŠçãè¡ãèœåãéèŠã§ãã
次ã«ãããŒã¿ã¢ãŒããã¯ãã¯ãäŒæ¥ã®ããŒã¿æŠç¥ãèšèšããããžã·ã§ã³ã§ãã圌ãã¯ããžãã¹ã®èŠæ±ãæºããããã«ãããŒã¿æ§é ãã€ã³ãã©ãèšç»ã»æ§ç¯ãã責任ãæ ããŸãããã®åœ¹å²ã¯ãé«åºŠãªæè¡çç¥èãšãšãã«ãäºæ¥ã®ç解ãæ±ããããéåžžã«æŠç¥çãªããžã·ã§ã³ã§ãã
ãŸããããç¹åããã¹ãã«ãå¿ èŠãªããã°ããŒã¿ãšã³ãžãã¢ã¯ãéæ§é åããŒã¿ããªã¢ã«ã¿ã€ã ããŒã¿ã®åŠçã«ç¹åããŠããŸãããã®ãããªããŒã¿ãšã³ãžãã¢ã¯ãApache HadoopãApache Sparkãªã©ã®ããã°ããŒã¿æè¡ã«é¢ããæ·±ãç¥èãæããŠããããšãäžè¬çã§ãã
ãã³ãã£ãŒäŒæ¥ãšå€§äŒæ¥ã®ã¡ãªããã»ãã¡ãªãã
ãã³ãã£ãŒäŒæ¥ã¯ãããŒã¿ãšã³ãžãã¢ã«ãšã£ãŠã®é åãšããŠãæè»ãªåãæ¹ãšè¿ éãªææ決å®ãæããããšãã§ããŸããå°èŠæš¡ã§ããããšãå€ãã圹å²ãå€æ§ã§ããŒã éã®é£æºãå¯ãªãããå¹ åºãçµéšãç©ãããšãã§ããŸãããŸãã補åããµãŒãã¹ã«å¯ŸããçŽæ¥çãªåœ±é¿åãæã€ããšãã§ãããããä»äºã«å¯Ÿããå å®æãåŸããããããããŸããã
äžæ¹ã§å€§äŒæ¥ã§ã¯ãè±å¯ãªãªãœãŒã¹ãšå®å®ããæ¥åç°å¢ãã¡ãªãããšããŠæããããŸãã倧äŒæ¥ã¯ãã°ãã°è€éãªããŒã¿ã·ã¹ãã ãæã£ãŠããããããæ±ãããšã§é«åºŠãªæè¡åã磚ãããšãã§ããŸãããŸãã倧äŒæ¥ç¹æã®æ§é åããããã£ãªã¢ãã¹ããããããå°æ¥ã®ãã£ãªã¢åœ¢æã«åœ¹ç«ã€ã§ãããã
ãããã倧äŒæ¥ã«ãããŠã¯ãçµç¹ã倧ãããªããã¡ãªããã«ææ決å®ãç ©éã«ãªãããšããå人ã®è²¢ç®ãèŠéããããããšãããç¹ã«çæãå¿ èŠã§ãããã³ãã£ãŒäŒæ¥ã§ã¯ãè³éã®äžè¶³ãããžãã¹ã®æç¶æ§ã«äžå®ãæ®ãããšããã¡ãªãããšããŠæããããŸãã
çµç¹ãšããŠã®ããŒã¿æŽ»çšæåã®éèŠæ§
çµç¹ã«ãããããŒã¿æŽ»çšæåã®éžæã¯ãããŒã¿ãšã³ãžãã¢ã«ãšã£ãŠå€§å€éèŠã§ããããŒã¿ãåºç€ãšããææ決å®ãç©æ¥µçã«è¡ãããç°å¢ã§ã¯ãããŒã¿ãšã³ãžãã¢ã®ã¹ãã«ãå°éç¥èãæ倧éã«æŽ»çšãããŸããããã¯ãäŒæ¥ã®ç«¶äºååäžã ãã§ãªããããŒã¿ãšã³ãžãã¢èªèº«ã®è·åæºè¶³åºŠã«ãšã£ãŠãéèŠãªèŠçŽ ã§ãã
ããŒã¿æŽ»çšæåãæã€çµç¹ã§ã¯ãããŒã¿ã«å¯Ÿããã¢ã¯ã»ã¹ã容æã§ããã瀟å ã§ã®ããŒã¿å ±æãä¿é²ããäœå¶ãæŽã£ãŠããããšãäžè¬çã§ããå ããŠããšã³ãžãã¢ã ãã§ãªãçµå¶å±€ãå«ãå šå¡ãããŒã¿ã®äŸ¡å€ãç解ããæ¯æããŠããããšãæåã®ããã«ã¯äžå¯æ¬ ã§ãã
æçµçã«ãããŒã¿ãšã³ãžãã¢ã¯äŒæ¥ã«ãããããŒã¿ç®¡çãšæŽ»çšããã»ã¹ã®å°é家ãšããŠãçµç¹å šäœã®ããŒã¿æŽ»çšèœååäžã«å¯äžãã圹å²ãæ ã£ãŠããŸãããã®ããã«ã¯ãæè¡çãªç¥èãšãšãã«ãããžãã¹ã®æé·ãä¿é²ããããã®ããŒã¿å©çšã®æ©äŒãåžžã«æ¢æ±ããææ¡ããŠãã姿å¢ãæ±ããããŸãã
転è·ãµã€ããšãšãŒãžã§ã³ãã®æŽ»çšè¡
ããŒã¿ãšã³ãžãã¢ãšããŠæ°ããªãã£ãªã¢ãæ¢æ±ããéã¯ã転è·ãµã€ããšãšãŒãžã§ã³ãã®æãã圹å²ãéåžžã«éèŠã§ããæå¹ã«ãããã®ããŒã«ã䜿ãããªãããã®æŠç¥ã身ã«ã€ããŸããããæåãžã®ç¬¬äžæ©ã¯ãé©åãªè»¢è·ãµã€ãã®éžæãšãã®äœ¿ãæ¹ãç解ããããšããå§ãŸããŸãã
転è·ãµã€ãã®éžã³æ¹ãšäœ¿ãæ¹
転è·ãµã€ããéžæããéã«ã¯ãæ²èŒãããŠããè·çš®ã®å¹ ãæ¥çãäŒæ¥ã®è³ªã«çç®ããå¿ èŠããããŸããããŒã¿ãšã³ãžãã¢ã«ç¹åããæ±äººãå€ããµã€ãããåæ¥çã®æåç·ã§åããŠããäŒæ¥ã®æ±äººãæ²èŒãããŠãããµã€ããéžã¶ãšè¯ãã§ãããã
䜿ãæ¹ãæ倧éã«æŽ»ããããã«ã¯ãèªåã®ã¹ãã«ã»ããããã£ãªã¢ãã©ã³ãæ確ã«ããããã«åã£ãæ±äººããã£ã«ã¿ãªã³ã°ããæ©èœãç©æ¥µçã«äœ¿çšããããšãèèŠã§ãããŸããã¬ãžã¥ã¡ãè·åçµæŽãäžå¯§ã«èšè¿°ããèªèº«ã®çµéšãã¢ããŒã«ã§ããããã«ããŸãããã
ããã«ãéžãã 転è·ãµã€ãã§å®æçã«ãããã£ãŒã«ãæŽæ°ããã¢ã¯ãã£ãã«æ±äººããã§ãã¯ããããšã倧åã§ãããµã€ãã«ãã£ãŠã¯ãã£ãªã¢ã¢ããã€ã¶ãŒãç¡æã§çžè«ã«ä¹ã£ãŠãããããããã®ãããªãµãŒãã¹ãå©çšããŠèªåã«åã£ãã¢ããã€ã¹ãåããããšãéèŠã§ãã
ãšãŒãžã§ã³ãã®éžã³æ¹ãšæ±ãããµããŒãå 容
è¯è³ªãªãšãŒãžã§ã³ãã®éžã³æ¹ãšããŠã¯ãç¹ã«ãã®ãšãŒãžã§ã³ããäŒç€ŸãããŒã¿ãšã³ãžãã¢ãªã³ã°åéã«è©³ãããã©ãããèŠæ¥µããããšãéèŠã§ããåžå Žã®ãã¬ã³ããç解ããŠããŠãããªãã®ãã£ãªã¢ç®æšã«ãããããäŒæ¥ãç¥ã£ãŠãããšãŒãžã§ã³ããæãŸããã§ãã
ãµããŒãå 容ã§ã¯ãå±¥æŽæžãè·åçµæŽæžã®ãã©ãã·ã¥ã¢ãããé¢æ¥å¯Ÿçã絊äžäº€æžã®ã¢ããã€ã¹ãªã©ãæ±ããããŸãããŸããããªãã®é·æçãªãã£ãªã¢èšç»ãç解ããããããµããŒãããæåã®ãããšãŒãžã§ã³ããéžã¶ããšã倧åã§ãã
ãšãŒãžã§ã³ããšã®é£æºã§ã¯ãå®æçãªé£çµ¡ãåãåãããšãæåãžã®ã«ã®ãæ¡ããŸãããšãŒãžã§ã³ããææ¡ããæ±äººãèªåã«ãšã£ãŠé©åãã©ãããè©äŸ¡ããå¿ èŠã«å¿ããŠãã£ãªã¢ç®æšã«é¢ããæèŠãäŒããããšã§ãããè¯ã転è·æŽ»åãå±éã§ããŸãã
çžè«ãã¹ããã€ã³ããšæ³šæç¹
転è·ãšãŒãžã§ã³ãããµã€ãã«çžè«ããéã«ã¯ãèªèº«ã®ãã£ãªã¢ãã©ã³ãšæåŸ ããè·åå 容ãã¯ã£ãããããããšãæ±ããããŸãããŸããçŸåšã®ã¹ãã«ãçµéšã ãã§ãªããå°æ¥çã«èº«ã«ã€ãããã¹ãã«ãçµéšã«ã€ããŠã話ãåããŸãããã
泚æç¹ãšããŠã¯ãæäŸãããæ å ±ãåžžã«èªåã«ãšã£ãŠæåãã©ãããå€æããå¿ èŠãããããšã§ããèªèº«ã®çŽæã転è·åžå Žã®ååãèŠãªããããšãŒãžã§ã³ãã®æèŠãåèã«ãã€ã€æçµæ±ºå®ãäžãããã«ããŠãã ããã
ããã«ã転è·ãµã€ãããšãŒãžã§ã³ãã«éãããåæ¥ä»ç€Ÿã®çŸåœ¹ããŒã¿ãšã³ãžãã¢ããã®æ å ±ãåŸãããšãéèŠã§ããå®éã®è·å Žã®ç¶æ³ãäŒæ¥æåã«ã€ããŠãªã¢ã«ãªæèŠãåŸãããšãã§ããã§ãããã
æåã®ããã®ãããã¯ãŒãã³ã°æŠç¥
ãããã¯ãŒãã³ã°ã¯ãããŒã¿ãšã³ãžãã¢ã®è»¢è·æåã«äžå¯æ¬ ãªèŠçŽ ã§ããæ¥çã®ã€ãã³ããã»ãããŒã«ç©æ¥µçã«åå ããåæ¥è ãæ¥çã®ãªãŒããŒãã¡ãšã®ã€ãªãããç¯ãããšã倧åã§ãã
ãªã³ã©ã€ã³ãã©ãããã©ãŒã ãå©çšããŠã®ãããã¯ãŒãã³ã°ãå¹æçã§ããLinkedInãGitHubãªã©ã§èªåã®å°éæ§ãã¢ããŒã«ããç©æ¥µçã«é¢é£ããã³ãã¥ããã£ã«åå ããŸãããããããã®æŽ»åã¯ãæªå ¬éã®æ±äººæ å ±ãåŸãããçŽæ¥ãªã¯ã«ãŒã¿ãŒã«ã¢ãããŒãããããã£ããã«ããªããŸãã
æåŸã«ãæ¢åã®ã³ãã¯ã·ã§ã³ã倧åã«ããããããçºå±ãããããšãå¿ããŠã¯ãããŸããã以ååããäŒç€Ÿã®ååãäžåžãæ¥çå ã§å°æ¬ããå°é家ãªã©ããããããããã¯ãŒã¯ãéããŠæ å ±ãæ©äŒãåŸãããšãã転è·ãæåã«å°ãéµãšãªãã§ãããã
ããŒã¿ãšã³ãžãã¢è»¢è·æåã®ããã®é¢æ¥å¯Ÿç
ããŒã¿ãšã³ãžãã¢ãšããŠã®è»¢è·ãæåãããã«ã¯ãé©åãªé¢æ¥å¯Ÿçãäžå¯æ¬ ã§ããæ¥çã§æ±ããããã¹ãã«ãç¥èã¯ãã¡ãããé¢æ¥å®ã®åã§ããããåŠäœã«ã¢ããŒã«ããããéèŠãªãã€ã³ããšãªããŸãã以äžã«ãé¢æ¥ã«ããã質åãžã®çãæ¹ãããŒããã©ãªãªã®äœææ³ã絊äžäº€æžã®ã¢ãããŒããããã³å ¥ç€Ÿåã®æ¥åå 容ãšåŸ éã®ç¢ºèªã«ã€ããŠè§£èª¬ããŸãã
é¢æ¥ã§æ±ãããã質åãšçãæ¹
ããŒã¿ãšã³ãžãã¢ã®é¢æ¥ã§ã¯ãå°éç¥èãçµéšã«é¢ãã質åãåããããšãå€ãã§ããããŒã¿ããŒã¹ç®¡çãããã°ã©ãã³ã°èšèªãããã°ããŒã¿ã®åãæ±ãçµéšãªã©ãæ±ããããã¹ãã«ã»ããã¯å€å²ã«ããããŸãã
çãæ¹ã«ã€ããŠã¯ãå ·äœçãªãããžã§ã¯ãçµéšã解決ããåé¡ã瀺ããªãããèªèº«ã®ã¹ãã«ããããããã説æããããšãæå¹ã§ãããŸããä»äºã®éçšã§ã©ã®ããã«ããŒã ãšååãããããŸãã¯ãªãŒããŒã·ãããçºæ®ããããšãã£ãäŸãæããããšããå°è±¡ã匷ãããŸãã
å ããŠãããŒã¿ã«åºã¥ããŠææ決å®ãè¡ã£ãäŸããæ¥åå¹çåã«è²¢ç®ããããŒã«ã®éçºãªã©ãããŒã¿ãšã³ãžãã¢ã®ã³ã¢ã³ã³ããã³ã·ãŒã瀺ãäºäŸãæºåããŠãããŸãããã
ããŒããã©ãªãªã®äœæãšå¹æçãªæŽ»çšæ³
ããŒããã©ãªãªã¯ãããŒã¿ãšã³ãžãã¢ãšããŠã®ããªãã®æè¡åãçµéšã瀺ãéèŠãªããŒã«ã§ãããããžã§ã¯ãã®ãµã³ãã«ãåæã¬ããŒãããœãããŠã§ã¢ãããŒã«ãžã®è²¢ç®ãªã©ãå®çžŸãšãªãè³æããŸãšããŸãããã
å¹æçã«æŽ»çšããããã«ã¯ãé¢æ¥ã®åã«èªåã®ããŒããã©ãªãªãæ éã«ã¬ãã¥ãŒããåãããžã§ã¯ãã§äœ¿çšããæè¡ãã¡ãœãããããã³åŸãããææãç°¡æœã«èª¬æã§ããããã«ããŠããããšã倧åã§ãããããŠãé¢æ¥äžã«ããŒããã©ãªãªãåç §ããªãã話ãå±éããããšã§ããããã§ãã·ã§ãã«ãšããŠã®å®åãå ·äœçã«ã¢ããŒã«ããããšãå¯èœãšãªããŸãã
ãŸãããªã³ã©ã€ã³äžã§ã¢ã¯ã»ã¹å¯èœãªããŒããã©ãªãªãäœæãããªã³ã¯ãå±¥æŽæžãè·åçµæŽæžã«å«ããããšããå§ãããŸããããã«ãããé¢æ¥å®ãäºåã«ããªãã®ã¹ãã«ãææ¡ãããããªããŸãã
絊äžäº€æžã®æŠç¥ãšã¿ã€ãã³ã°
絊äžã¯è»¢è·æŽ»åã«ãããŠéèŠãªèŠçŽ ã§ãã亀æžã«èšãåã«ãããªãã®ã¹ãã«ãçµéšã«èŠåã£ãåžå ŽäŸ¡å€ãç解ããŠããããšãéèŠã§ãã
亀æžã®ã¿ã€ãã³ã°ã¯ããªãã¡ãŒãåããŠãããäžè¬çã§ããããããé¢æ¥ã®ã©ã®æ®µéã§çµŠäžã«ã€ããŠèšåãããã¯äŒæ¥ã«ããç°ãªããããæ床ãæã£ãŠå¯Ÿå¿ããå¿ èŠããããŸãããªãã¡ãŒã¬ã¿ãŒãããã£ãåŸã«ãå ·äœçãªæ°åãæ¡ä»¶ã«ã€ããŠäº€æžãéå§ããã®ãç¡é£ã§ãã
æŠç¥çã«ã¯ãèªåã®äŸ¡å€ããã£ãããšã¢ããŒã«ãããªããã®å ±é ¬ãèŠæ±ããã®ããè£ä»ããæ å ±ãæºåããŠããããšãéèŠã§ãããŸããæè»æ§ãæã¡ã€ã€ããèªåã«ãšã£ãŠåãå ¥ããããæ¡ä»¶ã«ã¯æ¯ ç¶ãšããæ 床ã§èšãããšãæ±ããããŸãã
å ¥ç€Ÿåã«ç¢ºèªãã¹ãæ¥åå 容ãšåŸ é
å ¥ç€Ÿåã«æ¥åå 容ãšåŸ éããã£ãã確èªããããšã¯ãæªæ¥ã®ãã£ãªã¢ãã¹ãšè·å Žã§ã®æºè¶³åºŠãå·Šå³ããŸããå ·äœçãªæ¥åç¯å²ãããŒã ã®æ§æã䜿çšããæè¡ã¹ã¿ãã¯ããã£ãªã¢æé·ã®æ©äŒãªã©ã詳ããææ¡ããŠããããšãæãŸããã§ãã
ãŸããå€åæéãäŒæããã¬ãã¯ã¹ã¿ã€ã ããªã¢ãŒãã¯ãŒã¯ã®å¯åŠãä¿éºãçŠå©åçãªã©ã®åŸ éã«ã€ããŠãã詳现ã確èªããŠããããšããå§ãããŸãããããã®æ å ±ã¯ãããªããæäŸããããªãã¡ãŒãæ¯èŒãã決æãäžãéã«éèŠãªåºæºãšãªããŸãã
é¢æ¥ã®æçµæ®µéããªãã¡ãŒãåããéã«ããããã®ç¹ãã¯ãªã¢ã«ããããã®è³ªåãæºåããŠãããšè¯ãã§ããããããªããæ確ãªæåŸ ãæã£ãŠå ¥ç€Ÿã§ããããã«ãäºåã«èãã¹ãããšããªã¹ãã¢ããããŠããããšã§ãå ¥ç€ŸåŸã®ãã¹ããããé²ããå¯èœæ§ãé«ãŸããŸãã
ãŸãšã
ããŒã¿ãšã³ãžãã¢ã®è»¢è·ã¯ãããžãã¹ããŒãœã³ã«ãšã£ãŠæ³šç®ã®ãã£ãªã¢ãã§ã³ãžã§ããåºç€ç¥èãæè¡ã¹ãã«ã¯ãã¡ããããœããã¹ãã«ãéèŠãåžå ŽäŸ¡å€ã®é«ãŸããèæ¯ã«ãé²ãã¹ããã£ãªã¢ãã¹ãæ¢ããŸãããã転è·æŽ»åã§ã¯ãå±¥æŽæžã®æžãæ¹ãé¢æ¥æŠç¥ãæ¥çç 究ãæåã®éµã§ãã䜿çšãã転è·ãµã€ãããšãŒãžã§ã³ãã®éžå®ãå¿ããªãã§ãã ããããŸããããŒããã©ãªãªã®äœæã絊äžäº€æžã®ãã¯ããã¯ã圹ç«ã¡ãŸããããŒã¿ãšã³ãžãã¢ãšããŠé£èºããããã®å¹æçãªè»¢è·æ å ±ãããã«éçŽãããŠããŸãã