[젯슨나노 프로젝트]자율 주행 배달 로봇 Selly(8)_visual SLAM 테스트

우리 팀은 자율주행에 있어서 딥러닝 기반의 이미지 처리를 핵심 아이디어로 가져와 프로젝트를 진행하고 있다.

SLAM 관련 논문을 읽던 중, 기술 이름만 알고 있던 visual SLAM 기술에 대해 자세히 알게 되었고, vision 기술을 중시하는 우리 프로젝트의 특성상 visual SLAM을 사용하는 것이 좋겠다고 생각했다

 

visual SLAM은 SLAM에 비해서 최근에 연구되는 기술이라 오픈소스가 많이 없고 정보가 없어서 애먹었다.

그 중 xdspacelab에서 visual SLAM 기술을 연구하고 오픈소스로 제공하고 있다.

https://arxiv.org/pdf/1910.01122.pdf

(논문)

 

 

https://github.com/xdspacelab/openvslam

xdspacelab/openvslam

 

https://openvslam.readthedocs.io/en/master/overview.html

Overview — OpenVSLAM documentation

 

 

 

---

카메라 config 파일

---

Camera.name: "LUMIX DMC-GX8 8mm fisheye 1K"
Camera.setup: "monocular"
Camera.model: "fisheye"

Camera.fx: 441.730011169
Camera.fy: 442.520822476
Camera.cx: 480.35667528
Camera.cy: 275.490228646

Camera.k1: -3.068701296607466433e-02
Camera.k2: -3.343454364086094217e-03
Camera.k3: -2.881735840896060968e-03
Camera.k4: -5.917420310474077278e-04

Camera.fps: 30.0
Camera.cols: 960
Camera.rows: 540

Camera.color_order: "RGB"

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 2000
Feature.scale_factor: 1.2
Feature.num_levels: 8
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 7

#========================#
# Initializer Parameters #
#========================#

Initializer.scaling_factor: 5.0

fisheye 카메라 config file

 

Camera.name: "RICOH THETA S 960"
Camera.setup: "monocular"
Camera.model: "equirectangular"

Camera.fps: 30.0
Camera.cols: 1920
Camera.rows: 960

Camera.color_order: "RGB"

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 2000
Feature.scale_factor: 1.2
Feature.num_levels: 8
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 7
Feature.mask_rectangles:
  - [0.0, 1.0, 0.0, 0.1]
  - [0.0, 1.0, 0.84, 1.0]

360도 카메라 config file

 

Camera.name: "EuRoC monocular"
Camera.setup: "monocular"
Camera.model: "perspective"

Camera.fx: 458.654
Camera.fy: 457.296
Camera.cx: 367.215
Camera.cy: 248.375

Camera.k1: -0.28340811
Camera.k2: 0.07395907
Camera.p1: 0.00019359
Camera.p2: 1.76187114e-05
Camera.k3: 0.0

Camera.fps: 20.0
Camera.cols: 752
Camera.rows: 480

Camera.color_order: "Gray"

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 1000
Feature.scale_factor: 1.2
Feature.num_levels: 8
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 7

perspective 카메라 config file(흔히 생각하는 카메라인 것 같은데. zed는 이거로 쓰면 될 것 같다.)

 

Camera.name: "EuRoC stereo"
Camera.setup: "stereo"
Camera.model: "perspective"

# new "rectified" matrices is the first three cols of the projection matrix which is calculated with cv::stereoRectify()
# e.g. fx = P1[0][0] or P2[0][0], cx = P1[0][2] or P2[0][2]
#      fy = P1[1][1] or P2[1][1], cy = P1[1][2] or P2[1][2]

Camera.fx: 435.2046959714599
Camera.fy: 435.2046959714599
Camera.cx: 367.4517211914062
Camera.cy: 252.2008514404297

# there is no distortion after stereo rectification

Camera.k1: 0.0
Camera.k2: 0.0
Camera.p1: 0.0
Camera.p2: 0.0
Camera.k3: 0.0

# focal_x_baseline is -P2[0][3] which is calculated with cv::stereoRectify()

Camera.fps: 20.0
Camera.cols: 752
Camera.rows: 480
Camera.focal_x_baseline: 47.90639384423901

Camera.color_order: "Gray"

#======================#
# Stereo Rectification #
#======================#

# original intrinsic parameters (K, D) and stereo-recitification parameters (R)
# matrices (K, R) are written in row-major order

StereoRectifier.K_left: [458.654, 0.0, 367.215, 0.0, 457.296, 248.375, 0.0, 0.0, 1.0]
StereoRectifier.D_left: [-0.28340811, 0.07395907, 0.00019359, 1.76187114e-05, 0.0]
StereoRectifier.R_left: [0.999966347530033, -0.001422739138722922, 0.008079580483432283, 0.001365741834644127, 0.9999741760894847, 0.007055629199258132, -0.008089410156878961, -0.007044357138835809, 0.9999424675829176]
StereoRectifier.K_right: [457.587, 0.0, 379.999, 0.0, 456.134, 255.238, 0.0, 0.0, 1]
StereoRectifier.D_right: [-0.28368365, 0.07451284, -0.00010473, -3.555907e-05, 0.0]
StereoRectifier.R_right: [0.9999633526194376, -0.003625811871560086, 0.007755443660172947, 0.003680398547259526, 0.9999684752771629, -0.007035845251224894, -0.007729688520722713, 0.007064130529506649, 0.999945173484644]

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 1000
Feature.scale_factor: 1.2
Feature.num_levels: 8
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 7

#========================#
# Initializer Parameters #
#========================#

Initializer.num_min_triangulated_pts: 100

#===========================#
# PangolinViewer Parameters #
#===========================#

PangolinViewer.keyframe_size: 0.07
PangolinViewer.keyframe_line_width: 1
PangolinViewer.graph_line_width: 1
PangolinViewer.point_size: 2
PangolinViewer.camera_size: 0.08
PangolinViewer.camera_line_width: 3
PangolinViewer.viewpoint_x: 0
PangolinViewer.viewpoint_y: -0.65
PangolinViewer.viewpoint_z: -1.9
PangolinViewer.viewpoint_f: 400

stereo 카메라 config file(카메라 렌즈 두 개 이상)

​

openVSLAM 깃허브 example 폴더에 config파일이 몇 개 있는데, 그 중 perspective 카메라 보면 공통적으로 캘리브레이션 파라미터가 들어간다. 맞게 만든 듯!

 

Camera.name: "Selly_vision"
Camera.setup: "monocular"
Camera.model: "perspective"

Camera.fps: 25.0
Camera.cols: 480
Camera.rows: 270

Camera.color_order: "RGB"

Camera.fx: 537.021240234375
Camera.fy: 537.021240234375
Camera.cx: 623.784423828125
Camera.cy: 255.7088423046875
Camera.focal_x_baseline: 119.471

Camera.k1: 0.0
Camera.k2: 0.0
Camera.p1: 0.0
Camera.p2: 0.0
Camera.k3: 0.0

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 2000
Feature.scale_factor: 1.2
Feature.num_levels: 8
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 7

위 예시들을 보고 zed 카메라의 config file을 만들었다. 파라미터는 스테레오랩스에서 제공하는 카메라 파라미터&카메라 캘리브레이션 API로 쉽게 추출 가능하다.

https://www.stereolabs.com/docs/video/camera-calibration/

Camera Calibration | Stereolabs

https://www.stereolabs.com/docs/api/structsl_1_1CameraParameters.html

CameraParameters Struct Reference | API Reference | Stereolabs

 

 

작성한 zed 카메라의 캘리브레이션 파일로 openVSLAM을 작동시키면 이와같이 맵이 그려진다.

그려진 맵의 정보와 녹화된 비디오는 저장되고, localization 기능을 작동시키면 녹화한 비디오에서 feature를 추출하며 저장된 맵을 비교하여 위치를 추정한다.