Einstein is now a known tech in market and Salesforce now making its library public from the beta mode. Recently Salesforce made Einstein Object Detection public available. So Einstein Object Detection tells you where one or more objects are in an image which allows you to get specific counts and locations of objects within images.
Get Access Token
To make any API request, Firstly we need to generate Access token. So you can find detail steps for generating token here. For Object Detection first we need to train the Dataset. The process of dataset creation is bit different then the Einstein Vision.
So to train an Einstein Object Detection model, we must provide the coordinates of the bounding boxes around the objects we want detected. Draw a box around the object we want detected, this is called a bounding box. For training we need the x/y pixel coordinates of the top left edge of that box, as well as the pixel width and height. We can have multiple objects within a single example image.
Firstly we will collecte all of our example images and object data. Secondly we create a file with the name annotations.csv. This file is a key part of the Einstein Object Detection training. It tells the Einstein Platform where in our sample data it will find the objects it needs to pay attention to during training.
This is how the sample file will look. For the demo purpose I have used the same file used in Einstein Site. After the dataset is created you need to train It. This process is same as Einstein Vision. And once the dataset is trained you are ready to make your first request find the object details.
Create the dataset.
HttpRequest req = new HttpRequest();
req.setMethod('POST');
req.setEndpoint('https://api.einstein.ai/v2/vision/datasets/upload');
req.setHeader('content-type', 'multipart/form-data; charset="UTF-8"; boundary="1ff13444ed8140c7a32fc4e6451aa76d"');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setHeader('Cache-Control', 'no-cache');
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('path', 'https://einstein.ai/images/alpine.zip');
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('type', 'image-detection');
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
req.setBodyAsBlob(formBlob);
req.setHeader('Connection', 'keep-alive');
req.setHeader('Content-Length', contentLength);
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);
Map<String,object> responseMap =(Map<String,object>)JSON.deserializeUntyped(res.getBody()) ;
As this is a Asynchronous process so we need to check the status of dataset creation. So for this we will make another request.
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setEndpoint('https://api.einstein.ai/v2/vision/datasets/<DATASET ID>');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);
Train The Dataset
So once our dataset is created. we need to train the dataset.
HttpRequest req = new HttpRequest();
req.setMethod('POST');
req.setEndpoint('https://api.einstein.ai/v2/vision/train');
req.setHeader('content-type', 'multipart/form-data; charset="UTF-8"; boundary="1ff13444ed8140c7a32fc4e6451aa76d"');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setHeader('Cache-Control', 'no-cache');
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('name','Alpine Boxes on Shelves');
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('datasetId', '<DATASET ID>');
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
req.setBodyAsBlob(formBlob);
req.setHeader('Connection', 'keep-alive');
req.setHeader('Content-Length', contentLength);
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);
As this is also asynchronous process we we need to check the status.
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setEndpoint('https://api.einstein.ai/v2/vision/train/&amp;amp;lt;MODEL ID&amp;amp;gt;');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);Request to Detect Object
Now once our Model training is completed we are ready to make our request. So for testing purpose we will use this image.
HttpRequest req = new HttpRequest();
req.setMethod('POST');
req.setEndpoint('https://api.einstein.ai/v2/vision/detect');
req.setHeader('content-type', 'multipart/form-data; charset="UTF-8"; boundary="1ff13444ed8140c7a32fc4e6451aa76d"');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setHeader('Cache-Control', 'no-cache');
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('sampleLocation','https://newstechnologystuff.com/alpine.jpg');
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('modelId', '<MODEL ID>');
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
req.setBodyAsBlob(formBlob);
req.setHeader('Content-Length', contentLength);
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);So we got the final response. While I am not processing it, you can use JSON parser to read it.
{{ "probabilities": [ { "resultType": "DetectionResult", "label": "Alpine - Oat Cereal", "probability": 0.9947397, "boundingBox": { "minX": 2112, "minY": 916, "maxX": 2880, "maxY": 1933 } }, { "resultType": "DetectionResult", "label": "Alpine - Corn Flakes", "probability": 0.9945168, "boundingBox": { "minX": 791, "minY": 915, "maxX": 1540, "maxY": 1853 } }, { "resultType": "DetectionResult", "label": "Alpine - Bran Cereal", "probability": 0.9940328, "boundingBox": { "minX": 2873, "minY": 848, "maxX": 3629, "maxY": 1945 } }, { "resultType": "DetectionResult", "label": "Alpine - Bran Cereal", "probability": 0.9933943, "boundingBox": { "minX": 1433, "minY": 918, "maxX": 2156, "maxY": 1957 } } ]}
We got the correct result with almost 99% probability. So this is very important feature of Einstein, therefore can be used in real life in store to detect inventory, Parts etc.
So tell me in comments section what did you like most about Einstein Object Detection. Happy Programming 🙂