https://github.com/jsbroks/coco-annotator seems having very nice architectures, and convenient as a tool.
With python,
https://github.com/waspinator/pycococreator
https://github.com/jsbroks/imantics
are candidates.
https://imantics.readthedocs.io/en/latest/api.html#mask-object seems have good data structures with useful functions such as union()
For datasets
https://github.com/cocodataset/cocoapi
