Column Store for GWAC: A High-cadence, High-density, Large-scale Astronomical Light Curve Pipeline and Distributed Shared-nothing Database
文献类型：期刊论文
| 作者 | Wan, Meng1,2,3,4 ; Wu, Chao1,4; Wang, Jing1,4; Qiu, Yulei1,4; Xin, Liping1,4; Mullender, Sjoerd3; Muhleisen, Hannes3; Scheers, Bart3; Zhang, Ying3,5; Nes, Niels3
|
| 刊名 | PUBLICATIONS OF THE ASTRONOMICAL SOCIETY OF THE PACIFIC
 |
| 出版日期 | 2016-11-01 |
| 卷号 | 128期号:969 |
| 关键词 | astronomical databases: miscellaneous catalogs |
| 英文摘要 | The ground-based wide-angle camera array (GWAC), a part of the SVOM space mission, will search for various types of optical transients by continuously imaging a field of view (FOV) of 5000 degrees(2) every 15 s. Each exposure consists of 36 x 4k x 4k pixels, typically resulting in 36 x similar to 175,600 extracted sources. For a modern time-domain astronomy project like GWAC, which produces massive amounts of data with a high cadence, it is challenging to search for short timescale transients in both real-time and archived data, and to build long-term light curves for variable sources. Here, we develop a high-cadence, high-density light curve pipeline (HCHDLP) to process the GWAC data in real-time, and design a distributed shared-nothing database to manage the massive amount of archived data which will be used to generate a source catalog with more than 100 billion records during 10 years of operation. First, we develop HCHDLP based on the column-store DBMS of MonetDB, taking advantage of MonetDB's high performance when applied to massive data processing. To realize the real-time functionality of HCHDLP, we optimize the pipeline in its source association function, including both time and space complexity from outside the database (SQL semantic) and inside (RANGE-JOIN implementation), as well as in its strategy of building complex light curves. The optimized source association function is accelerated by three orders of magnitude. Second, we build a distributed database using a two-level time partitioning strategy via the MERGE TABLE and REMOTE TABLE technology of MonetDB. Intensive tests validate that our database architecture is able to achieve both linear scalability in response time and concurrent access by multiple users. In summary, our studies provide guidance for a solution to GWAC in real-time data processing and management of massive data. |
| WOS标题词 | Science & Technology ; Physical Sciences |
| 类目[WOS] | Astronomy & Astrophysics |
| 研究领域[WOS] | Astronomy & Astrophysics |
| 关键词[WOS] | TRANSIENTS ; TELESCOPE ; ARRAY ; LOFAR |
| 收录类别 | SCI |
| 语种 | 英语 |
| WOS记录号 | WOS:000387113900004 |
| 源URL | [http://ir.bao.ac.cn/handle/114a11/6858]  |
| 专题 | 国家天文台_空间科学研究部 |
| 作者单位 | 1.Chinese Acad Sci, Natl Astron Observ, 20A Datun Rd, Beijing 100012, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Ctr Wiskunde & Informat, NL-1098 XG Amsterdam, Netherlands 4.Chinese Acad Sci, Natl Astron Observ, Key Lab Space Astron & Technol, Beijing 100012, Peoples R China 5.MonetDB Solut, NL-1098 XG Amsterdam, Netherlands 6.Capital Univ Econ & Business, Coll Econ, Beijing 100070, Peoples R China |
| 推荐引用方式 GB/T 7714 | Wan, Meng,Wu, Chao,Wang, Jing,et al. Column Store for GWAC: A High-cadence, High-density, Large-scale Astronomical Light Curve Pipeline and Distributed Shared-nothing Database[J]. PUBLICATIONS OF THE ASTRONOMICAL SOCIETY OF THE PACIFIC,2016,128(969). |
| APA | Wan, Meng.,Wu, Chao.,Wang, Jing.,Qiu, Yulei.,Xin, Liping.,...&Wu,Chao.(2016).Column Store for GWAC: A High-cadence, High-density, Large-scale Astronomical Light Curve Pipeline and Distributed Shared-nothing Database.PUBLICATIONS OF THE ASTRONOMICAL SOCIETY OF THE PACIFIC,128(969). |
| MLA | Wan, Meng,et al."Column Store for GWAC: A High-cadence, High-density, Large-scale Astronomical Light Curve Pipeline and Distributed Shared-nothing Database".PUBLICATIONS OF THE ASTRONOMICAL SOCIETY OF THE PACIFIC 128.969(2016). |
入库方式： OAI收割
来源：国家天文台
浏览0
下载0
收藏0
其他版本
除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。