Object detection on multiple datasets with an automatically learned unified label space.

Overview

Simple multi-dataset detection

An object detector trained on multiple large-scale datasets with a unified label space; Winning solution of ECCV 2020 Robust Vision Challenges.

Simple multi-dataset detection,
Xingyi Zhou, Vladlen Koltun, Philipp Krähenbühl,
arXiv technical report (arXiv 2102.13086)

Contact: [email protected]. Any questions or discussions are welcomed!

Abstract

How do we build a general and broad object detection system? We use all labels of all concepts ever annotated. These labels span diverse datasets with potentially inconsistent taxonomies. In this paper, we present a simple method for training a unified detector on multiple large-scale datasets. We use dataset-specific training protocols and losses, but share a common detection architecture with dataset-specific outputs. We show how to automatically integrate these dataset-specific outputs into a common semantic taxonomy. In contrast to prior work, our approach does not require manual taxonomy reconciliation. Our multi-dataset detector performs as well as dataset-specific models on each training domain, but generalizes much better to new unseen domains. Entries based on the presented methodology ranked first in the object detection and instance segmentation tracks of the ECCV 2020 Robust Vision Challenge.

Features at a glance

  • We trained a unified object detector on 4 large-scale detection datasets: COCO, Objects365, OpenImages, and Mapillary, with state-of-the-art performance on all of them.

  • The model predicts class labels in a learned unified label space.

  • The model can be directly used to test on novel datasets outside the training datasets.

  • In this repo, we also provide state-of-the-art baselines for Objects365 and OpenImages.

Main results

COCO test-challenge OpenImages public test Mapillary test Objects365 val
52.9 60.6 25.3 33.7

Results are obtained using a Cascade-RCNN with ResNeSt200 trained in an 8x schedule.

  • Unified model vs. ensemble of dataset-specific models with known test domains.
COCO Objects365 OpenImages mean.
Unified 45.4 24.4 66.0 45.3
Dataset-specific models 42.5 24.9 65.7 44.4

Results are obtained using a Cascade-RCNN with Res50 trained in an 8x schedule.

  • Zero-shot cross dataset evaluation
VOC VIPER CityScapes ScanNet WildDash CrowdHuman KITTI mean
Unified 82.9 21.3 52.6 29.8 34.7 70.7 39.9 47.3
Oracle models 80.3 31.8 54.6 44.7 - 80.0 - -

Results are obtained using a Cascade-RCNN with Res50 trained in an 8x schedule.

Installation

Our project is developed on detectron2. Please follow the official detectron2 installation. All our code is under projects/UniDet/. In theory, you should be able to copy-paste projects/UniDet/ to the latest detectron2 release or your own detectron2 repo to run our project. There might be API changes in future detectron2 releases that make it incompatible.

Demo

We use the same inference API as detectorn2. To run inference on an image folder using our pretrained model, run

python projects/UniDet/demo/demo.py --config-file projects/UniDet/configs/Unified_learned_OCIM_R50_6x+2x.yaml --input images/*.jpg --opts MODEL.WEIGHTS models/Unified_learned_OCIM_R50_6x+2x.pth

If setup correctly, the output should look like:

*The sample image is from WildDash dataset.

Note that the model predicts all labels in its label hierarchy tree (for example, both vehicle and car for a car), following the protocol in OpenImages.

Benchmark evaluation and training

After installation, follow the instructions in DATASETS.md to setup the (many) datasets. Then check REPRODUCE.md to reproduce the results in the paper.

License

All our code under projects/Unidet/ is under Apache 2.0 license. The code from detectron2 follows the original Apache 2.0 license.

Citation

If you find this project useful for your research, please use the following BibTeX entry.

@inproceedings{zhou2021simple,
  title={Simple multi-dataset detection},
  author={Zhou, Xingyi and Koltun, Vladlen and Kr{\"a}henb{\"u}hl, Philipp},
  booktitle={arXiv preprint arXiv:2102.13086},
  year={2021}
}
Comments
  • result type Float can't be cast to the desired output type Long

    result type Float can't be cast to the desired output type Long

    when i use 1 GPU(v100) "python projects/UniDet/train_net.py --config-file projects/UniDet/configs/Partitioned_COI_R50_2x.yaml --num-gpus 1",batch=16, there is no problem.

    When I use two GPUs( 2x v100), an error occurred

    python projects/UniDet/train_net.py --config-file projects/UniDet/configs/Partitioned_COI_R50_2x.yaml --num-gpus 1",batch=32

    image

    How can I solve this problem? Thank you!

    opened by ahong007007 2
  • Could you provide the files in the label_space folder

    Could you provide the files in the label_space folder

    Hi Xingyi,

    Thanks for you wonderful and easy to use work! I have managed to do some testing with the models and they performed quite well.

    However I am not able to find the files under label_spaces folder to train, e.g., datasets/label_spaces/learned_mAP+M.json Could you provide those?

    Also as a suggestion, maybe you can put the model_zoo in the main read_me. They are quite hard to find at a glance.

    Thanks

    • Links to the relevant documentation/comment:
    documentation 
    opened by Lancelot365 2
  • 6x+2x meaning

    6x+2x meaning

    📚 Documentation

    • Links to the relevant documentation/comment: Hello, Thanks for your work again. I have a question regarding the config file with '6x+2x'. The question is about the different training strategy during different stage, i.e., 6x stage and 2x stage. As I can see from the config file, The difference may be the use of EQL_LOSS; however, this loss depends the file 'datasets/oid/annotations/openimages_challenge_2019_train_v2_cat_info.json', which I do not find. Can you elaborate this a little bit? Many thanks. @xingyizhou
    documentation 
    opened by LigZhong 1
  • can't download preprocessed annotations

    can't download preprocessed annotations

    📚 Documentation

    The preprocessed annotations link is invalid, could you please reupload them? https://drive.google.com/drive/folders/1dDLiQfjEE0PqRlb7gtja-ermW2HJK5Tz?usp=sharing

    documentation 
    opened by biscuit279 0
  • question on the model settings in Tab. 5

    question on the model settings in Tab. 5

    Hi Xingyi, thanks for your wonderful work! I have a question regarding the model setting, in Tab. 5, for the naive merge, how is the merge conducted? and for the retrained, in my understanding, the box labels of multiple datasets are converted to the unified space and used for training a unified model, but still, the objects in one dataset may not be labeled in another, so we could still face such background confusion in supervision? is my understanding correct?

    Thanks in advance!

    opened by twangnh 1
  • Question about RPN

    Question about RPN

    Hello, I am very interested in your work, but I have a question to ask you. In the RPN stage, a proposal may be a positive sample in dataset A but a negative sample in dataset B. Will this produce ambiguity? Thanks!

    documentation 
    opened by hantaotao 1
  • Mapillary object detection dataset

    Mapillary object detection dataset

    Thank you for that great work. I would like to know which dataset of mapillary datasets has you used to train your model. I have check the website and I am not sure which one contains the object detection bounding boxes.

    Could you please provide a link to download the dataset.

    documentation 
    opened by Moaaz-Abdelrahman 0
  • Why some labels are duplicated in the unified detector?

    Why some labels are duplicated in the unified detector?

    For example, nightstand appears twice in the labels file learned_mAP.csv:

    line255: _nightstand_,,,nightstand,
    line693: Nightstand__,/m/02z51p,Nightstand,,
    

    But some labels from different datasets are normalized to one label. Why? Is it due to the When there are different label granularities, we keep them all in our label-space, and expect to predict all of them in the paper?

    documentation 
    opened by youyuge34 1
  • import error 'FastRCNNOutputs' from 'detectron2.modeling.roi_heads.fast_rcnn'

    import error 'FastRCNNOutputs' from 'detectron2.modeling.roi_heads.fast_rcnn'

    $ python projects/UniDet/demo/demo.py --config-file projects/UniDet/configs/Unified_learned_OCI_R50_6x.yaml --input images/34501842524_3c858b3080_k.jpg --opts MODEL.WEIGHTS models/Unified_learned_OCI_R50_6x.pth
    Traceback (most recent call last):
      File "projects/UniDet/demo/demo.py", line 18, in <module>
        from unidet.config import add_unidet_config
      File "projects/UniDet/unidet/__init__.py", line 17, in <module>
        from .modeling.roi_heads.custom_roi_heads import CustomROIHeads, CustomCascadeROIHeads
      File "projects/UniDet/unidet/modeling/roi_heads/custom_roi_heads.py", line 22, in <module>
        from .custom_fast_rcnn import CustomFastRCNNOutputLayers
      File "projects/UniDet/unidet/modeling/roi_heads/custom_fast_rcnn.py", line 16, in <module>
        from detectron2.modeling.roi_heads.fast_rcnn import FastRCNNOutputLayers, FastRCNNOutputs
    ImportError: cannot import name 'FastRCNNOutputs' from 'detectron2.modeling.roi_heads.fast_rcnn' (/ProjectRoot/pyprojects/venv/test2/lib/python3.7/site-packages/detectron2/modeling/roi_heads/fast_rcnn.py)
    

    I install detectron2 v0.6 built from https://dl.fbaipublicfiles.com/detectron2/wheels/cu101/torch1.8/index.html . The reason is detectron2 change its codebase, while ours is older one. So i follow the https://issueexplorer.com/issue/facebookresearch/unbiased-teacher/44 to fix the bug perfectly. I hope this can help someone who faces it.

    opened by youyuge34 2
Owner
Xingyi Zhou
CS Ph.D. student at UT Austin.
Xingyi Zhou
Go implementation of the yolo v3 object detection system

Go YOLO V3 This repository provides a plug and play implementation of the Yolo V3 object detection system in Go, leveraging gocv. Prerequisites Since

Wim Spaargaren 58 Dec 14, 2022
A high performance go implementation of Wappalyzer Technology Detection Library

wappalyzergo A high performance port of the Wappalyzer Technology Detection Library to Go. Inspired by https://github.com/rverton/webanalyze. Features

ProjectDiscovery 378 Jan 8, 2023
A tool for building identical machine images for multiple platforms from a single source configuration

Packer Packer is a tool for building identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs o

null 2 Oct 3, 2021
A high-performance concurrent scanner written by go, which can be used for survival detection, tcp port detection, and web service detection.

aScan A high-performance concurrent scanner written by go, which can be used for survival detection, tcp port detection, and web service detection. Fu

seventeen 16 Aug 15, 2022
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

End-to-end computer vision platform Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises. onepa

Onepanel, Inc. 642 Dec 12, 2022
An encrypted object storage system with unlimited space backed by Telegram.

TGStore An encrypted object storage system with unlimited space backed by Telegram. Please only upload what you really need to upload, don't abuse any

The golang.design Initiative 77 Nov 28, 2022
A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). Like Docker, but for data.

AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. You can think of it like Docker, but f

mya 8 Feb 9, 2022
Stash large datasets on GitHub for free, quick, and easy download 🐿

squirrel Stash large datasets on GitHub for free, quick, and easy download ?? Install To install squirrel, run the following: curl ... Usage Create a

wwm 0 Dec 31, 2021
Fast face detection, pupil/eyes localization and facial landmark points detection library in pure Go.

Pigo is a pure Go face detection, pupil/eyes localization and facial landmark points detection library based on Pixel Intensity Comparison-based Objec

Endre Simo 3.9k Dec 24, 2022
Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Bytedance Inc. 1.6k Dec 30, 2022
S3pd - CLI utility that downloads multiple s3 objects at a time, with multiple range-requests issued per object

S3 Parallel Downloader CLI utility that downloads multiple s3 objects at a time,

Colin Bookman 2 May 13, 2022
gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU

gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU is spending time on, displaying stack traces of your processes across native programs1 (includes Golang), Java and Python runtimes, and kernel routines.

Granulate 598 Dec 27, 2022
Go implementation of the yolo v3 object detection system

Go YOLO V3 This repository provides a plug and play implementation of the Yolo V3 object detection system in Go, leveraging gocv. Prerequisites Since

Wim Spaargaren 58 Dec 14, 2022
A proxy that authorizes and enforces a given label in a given PromQL query

prom-authzed-proxy prom-authzed-proxy is a proxy for Prometheus that authorizes the request's Bearer Token with Authzed and enforces a label in a Prom

authzed 30 Jul 19, 2022
Drone plugin to create comment and label in PR to Gitee

drone-plugin-gitee-pulls 中文文档 Drone plugin to create comment and label in PR to

kit101 4 Sep 6, 2022
Rpcx-framework - An RPC microservices framework based on rpcx, simple and easy to use, ultra fast and efficient, powerful, service discovery, service governance, service layering, version control, routing label registration.

RPCX Framework An RPC microservices framework based on rpcx. Features: simple and easy to use, ultra fast and efficient, powerful, service discovery,

ZYallers 1 Jan 5, 2022
Troon-NFT-Contract is deployed on Flow Blockchain, which is a white-label smart-contract for NFTs with an addition layer of Brand, Schema and Template

Overview Summary of NFTContract NFTContract is a Non Fungible Token (NFT) standard for Flow blockchain. It offers a powerful set while keeping unneces

null 0 Jan 4, 2022
A simple go tool, that calculates the allocated resources from all nodes matching the label selector.

kube-allocated-resources This is a simple go tool, that calculates the allocated resources from all nodes matching the label selector. Build Build on

Yusuf Kör 1 Jan 12, 2022
Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Themis provides strong, usable cryptography for busy people General purpose cryptographic library for storage and messaging for iOS (Swift, Obj-C), An

Cossack Labs 1.6k Jan 9, 2023
Uniqush is a free and open source software system which provides a unified push service for server side notification to apps on mobile devices.

Homepage Download Blog/News @uniqush Introduction Uniqush (\ˈyü-nə-ku̇sh\ "uni" pronounced as in "unified", and "qush" pronounced as in "cushion") is

Uniqush 1.5k Jan 9, 2023