3D Cal: An open-source software library for calibrating tactile sensors

To make sure you have all the necessary Python libraries, run the following line in your terminal:

pip3 install py3DCal

You can then import the library in your Python scripts:

import py3DCal as p3d

Below is a minimal example of how to use py3DCal to collect probe data.

Note: Replace the /dev/ttyUSB0 with the appropriate serial port for the Ender3. If you don't know the name of the port, the serial ports can be printed using p3d.list_com_ports() or typing list-com-ports in the terminal.

DIGIT:

import py3DCal as p3d

ender3 = p3d.Ender3("/dev/ttyUSB0")
digit = p3d.DIGIT("D20966")
calibrator = p3d.Calibrator(printer=ender3, sensor=digit)
calibrator.probe()

Replace the "D20966" with the serial number of your DIGIT sensor.

GelSight Mini:

import py3DCal as p3d

ender3 = p3d.Ender3("/dev/ttyUSB0")
gsmini = p3d.GelsightMini()
calibrator = p3d.Calibrator(printer=ender3, sensor=gsmini)
calibrator.probe()

Before using the GelSight Mini, make sure you have the GelSight SDK installed.

Connect to the printer

ender3.connect()

Homing the printer

ender3.initialize(xy_only=False)

Parameters:

• xy_only=False (default): Home the XYZ axes of the printer.
• xy_only=True: Only home the XY axes of the printer.

Moving the printer

ender3.go_to(x=5, y=5, z=2)

Parameters:

• x (optional): The desired x-coordinate of the print head.
• y (optional): The desired y-coordinate of the print head.
• z (optional): The desired z-coordinate of the print head.

Note: Any of the above parameters can be omitted if you don't want to move certain axes.

Sending G-Code (Ender 3) to the printer

ender3.send_gcode(command="G28")

Parameters:

• command: The G-Code (Ender 3) command to be sent to the printer.

Getting response from the printer

ender3.get_response()

Disconnect from printer

ender3.disconnect()

Connect to the sensor

digit.connect()

Taking an image with the sensor

digit.capture_image()

Disconnect from the sensor

digit.disconnect()

To add a new 3D printer, extend the Printer class and implement the following 6 functions:

def __init__(self):
    # Code to initialize printer variables

def connect(self):
    # Code to connect to the printer

def disconnect(self):
    # Code to disconnect from the printer

def send_gcode(self, command):
    # Code to execute G-Code (Ender 3) command on the printer

def get_response(self):
    # Code to return message from the printer

def initialize(self, xy_only=False):
    # Code to initialize printer (home, set_units, etc)

Example implementation for Ender3:

from p3d import Printer
import serial

class Ender3(Printer):
    def __init__(self, port):
        self.port = port
        self.name = "Ender 3"

    def connect(self):
        # Code to connect to the printer
        self.ser = serial.Serial(self.port, 115200)

    def disconnect(self):
        # Code to disconnect from the printer
        self.ser.close()

    def send_gcode(self, command):
        # Code to execute G-Code (Ender 3) command on the printer
        self.ser.write(str.encode(command + "\r\n"))

    def get_response(self):
        # Code to return message from the printer
        reading = self.ser.readline().decode('utf-8')

        return reading

    def initialize(self, xy_only=False):
        # Code to initialize printer (home, set units, set absolute/relative movements, adjust fan speeds, etc.)

        # Use Metric Values
        self.send_gcode("G21")

        # Absolute Positioning
        self.send_gcode("G90")

        # Fan Off
        self.send_gcode("M107")

        if xy_only:
            # Home Printer X Y
            self.send_gcode("G28 X Y")
        else:
            # Home Printer X Y Z
            self.send_gcode("G28")

        # Check if homing is complete
        ok_count = 0

        while ok_count < 4:
            if "ok" in self.get_response():
                ok_count += 1

        return True

To add a new tactile sensor, extend the Sensor class and implement the following 4 functions:

def __init__(self):
    # Sensor name
    self.name = "DIGIT"

    # The printer's x, y, and z coordinates at the sensor's origin
    self.x_offset = ...
    self.y_offset = ...
    self.z_offset = ...

    # How high the printer should move above the sensor before moving sideways
    self.z_clearance = ...

    # Maximum penetration for the sensor. The printer will not push into the sensor more than this value.
    self.max_penetration = ...

def connect(self):
    # Code to connect to the sensor

def disconnect(self):
    # Code to disconnect from the sensor

def capture_image(self):
    # Code to return an image from the sensor

Example implementation for DIGIT:

from tactile_calibration import Sensor
from digit_interface import Digit
import cv2

class GelsightDigit(Sensor):
    def __init__(self, serial_number):
        self.serial_number = serial_number
        self.name = "DIGIT"
        self.x_offset = 110
        self.y_offset = 111.5
        self.z_offset = 137
        self.z_clearance = 2
        self.max_penetration = 4
        self.default_calibration_file = "calibration_procs/digit/default.csv"

    def connect(self):
        # Code to connect to the sensor
        self.sensor = Digit(self.serial_number)
        self.sensor.connect()
        self.sensor.set_fps(30)
        
    def disconnect(self):
        # Code to disconnect from the sensor
        self.sensor.disconnect()

    def capture_image(self):
        # Code to return an image from the sensor
        return cv2.flip(self.sensor.get_frame(), 1)

To calculate the mapping from printer coordinates to pixel space, you only need to fit circles on 2 images. Annotations for the rest of the data will be automatically computed. To perform the circle fitting, use the following function:

import py3DCal as p3d

p3d.annotate(dataset_path="path/to/your/sensor_calibration_data", probe_radius_mm=2)

Performing the annotation is a 3 step process:

1. Fit the circle to the first image.
2. Fit the circle to the second image.
3. This screen shows you a sample of the circle fitting for 9 randomly selected images in your dataset. To make minor adjustments, you can increase/decrease the pixel-to-mm ratio using the r and f keys. You can also shift all of the circles up/down/left/right using the arrow keys.

To visualize the annotations (i.e., circle fitting), you can use the following function:

import py3DCal as p3d

p3d.visualize_annotations(dataset_path="path/to/your/sensor_calibration_data")

Parameters:

• dataset_path: The path to your sensor calibration data directory.
• img_idxs (optional): A tuple or list of image indices to visualize. By default, shows 3 random images.
• save_path (optional): The path to save the visualized annotations. If not specified, the visualizations will be displayed but not saved.

The following is a minimal example of how to train a TouchNet model from scratch using the data collected in Step 4:

import py3DCal as p3d
from py3DCal import datasets, models

# Create the dataset
my_dataset = datasets.TactileSensorDataset(root="./sensor_calibration_data")

# Create the model
touchnet = models.TouchNet()

# Train the model
p3d.train_model(model=touchnet, dataset=my_dataset, device="cuda")

# Generate depth maps using the trained model
depthmap = p3d.get_depthmap(model=touchnet,
  image_path="path/to/your/image.png",
  blank_image_path="path/to/your/blank_image.png",
  device="cuda"
)

Parameters for py3DCal.datasets.TactileSensorDataset():

• root: Root directory of the dataset.
• add_coordinate_embeddings (optional): Determines whether to add xy coordinate embeddings to the input images. Default is True.
• subtract_blank (optional): Determines whether to subtract the blank image from the input images. Default is True.
• transform (optional): Apply custom transformations to the input images. Default is torchvision.transforms.ToTensor().

Parameters for py3DCal.train_model():

• model: A torch.nn.Module object.
• dataset: A py3DCal.datasets.TactileSensorDataset object.
• num_epochs (optional): The number of epochs to train for. Default is 60.
• batch_size (optional): The batch size to use for training. Default is 64.
• learning_rate (optional): The learning rate to use for training. Default is 1e-4.
• train_ratio (optional): The split ratio for train and test sets. Default is 0.8.
• loss_fn (optional): The loss function to use for training. Default is nn.MSELoss().
• device (optional): Compute device to use ("cuda", "mps", "cpu" ). Default is "cpu".

To load a pretrained model, set load_pretrained=True when creating the TouchNet model:

DIGIT:

import py3DCal as p3d
from py3DCal import models

touchnet = models.TouchNet(
  load_pretrained=True,
  sensor_type=p3d.SensorType.DIGIT
  root="."
)

GelSight Mini:

import py3DCal as p3d
from py3DCal import models

touchnet = models.TouchNet(
  load_pretrained=True,
  sensor_type=p3d.SensorType.GELSIGHTMINI
  root="."
)

Parameters:

• load_pretrained (optional): Whether to load a pretrained model. Default is False
• sensor_type (optional if load_pretrained=False): If loading a pretrained model, choose the sensor type:
  • p3d.SensorType.DIGIT: Use for the DIGIT sensor.
  • p3d.SensorType.GELSIGHTMINI: Use for the GelSight Mini sensor.
• root (optional): Root directory for storing the downloaded pretrained weights (or loading them if they've already been downloaded). Defaults to the current working directory.

To load our dataset for the DIGIT (36,630 images) or GelSight Mini (36,270 images), use the following code:

DIGIT:

import py3DCal as p3d
from py3DCal import datasets

dataset = datasets.DIGIT(root='.', download=True)

GelSight Mini:

import py3DCal as p3d
from py3DCal import datasets

dataset = datasets.GelsightMini(root='.', download=True)

Parameters:

• root: The root directory where the dataset should be stored, or the root directory containting the digit_calibration_data or gsmini_calibration_data folders if the data has already been downloaded. Defaults to the current working directory.
• download (optional): Whether to download the dataset if it is not already present in the root directory. Default is False.
• add_coordinate_embeddings (optional): Whether to add coordinate embeddings to the dataset. Default is True.
• subtract_blank (optional): Whether to subtract the blank image from the dataset. Default is True.
• transform (optional): A transform to apply to the dataset. Default is torchvision.transforms.ToTensor().

TouchNet is a fully convolutional neural network designed to extract depth information from sensor readings. It takes as input an RGB tactile image augmented with a 2-channel x,y coordinate embedding, which provides the model with spatial context.

• The first channel encodes column indices as increasing pixel values across each row.
• The second channel encodes row indices as increasing pixel values across each column.

As shown in the figure below, TouchNet follows a 9-layer convolutional architecture that expands the 5-channel input to 256 channels, then contracts it to a 2-channel gradient map. The output channels correspond to the gradient in the x-direction (G_x) and gradient in the y-direction (G_y). Each module consists of a convolutional layer, batch normalization, a ReLU activation, and spatial dropout for regularization.