Salesforce Einstein Vision is part of the Einstein Platform Services technologies, and can used to easily AI-enable our apps with image recognition. In other words, we can build deep learning models for every use case including visual search, brand detection, and object identification with Einstein Vision. So we can leverage pre-trained classifiers, or train your own custom classifiers to solve a vast array of specialized image-recognition use cases. Developers can bring the power of image recognition to CRM and third-party applications so that end users across sales, service, and marketing can discover new insights about their customers and predict outcomes that lead to smarter decisions.
Salesforce Einstein Vision includes these APIs:
- Einstein Image Classification—Enables developers to train deep learning models to recognize and classify images at scale.
- Einstein Object Detection (Pilot)—Enables developers to train models to recognize and count multiple distinct objects within an image, providing granular details like the size and location of each object.
- Einstein OCR (Optical Character Recognition)—Use OCR models to detect alphanumeric text in an image.
So today we will cover the Salesforce Einstein Vision Classification. As we have a very good Apex wrapper provided by Salesforce dev team here. So we will use it to make all request of Einstein. While it handle all the heavy work in background for you.
But if you don’t want to install a full library to test these requests then I will share quick code sample which you can use to create your own request.
Before we start first you need to make sure that you have completed all prerequisite steps. Because we need a valid Einstein token to call API and few base classes to make request.
After all this setup our org is ready to make our first prediction. So for our demo purpose we will use Bike vs Car model. Here I have commented Token Id, Dataset Id and Model Id you can enter your related Id there.
Request for Access Token
Firstly, to make a request we need the access token, Here we take help of our base classes which we have included. So to get the access token we will use JWT access token helper.
public String getAccessToken() {
// Ignore the File upload part and "jwt.pkcs" if you used a Salesforce certificate to sign up
// for an Einstein Platform account
ContentVersion base64Content = [SELECT Title, VersionData FROM ContentVersion where Title='einstein_platform' OR Title='predictive_services' ORDER BY Title LIMIT 1];
String keyContents = base64Content.VersionData.tostring();
keyContents = keyContents.replace('-----BEGIN RSA PRIVATE KEY-----', '');
keyContents = keyContents.replace('-----END RSA PRIVATE KEY-----', '');
keyContents = keyContents.replace('\n', '');
// Get a new token
JWT jwt = new JWT('RS256');
// jwt.cert = 'JWTCert'; // Uncomment this if you used a Salesforce certificate to sign up for an Einstein Platform account
jwt.pkcs8 = keyContents; // Comment this if you are using jwt.cert
jwt.iss = 'developer.force.com';
jwt.sub = '<Your Email Address>';
jwt.aud = 'https://api.einstein.ai/v2/oauth2/token';
jwt.exp = '3600';
String access_token = JWTBearerFlow.getAccessToken('https://api.einstein.ai/v2/oauth2/token', jwt);
return access_token;
}Create Dataset
A data set is a folder which contains the images. Here we pass the data in multipart/form-data format.
HttpRequest req = new HttpRequest();
req.setMethod('POST');
req.setEndpoint('https://api.einstein.ai/v2/vision/datasets/upload');
req.setHeader('content-type', 'multipart/form-data; charset="UTF-8"; boundary="1ff13444ed8140c7a32fc4e6451aa76d"');
req.setHeader('Authorization', 'Bearer <TOKEN>'); //replace token with your access token
req.setHeader('Cache-Control', 'no-cache');
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('path', 'https://drive.google.com/uc?export=download&id=0BxAR1M3HgBCUTFY4YnlNdldpT00');
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('type', 'image');
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
req.setBodyAsBlob(formBlob);
req.setHeader('Connection', 'keep-alive');
req.setHeader('Content-Length', contentLength);
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);Dataset can be created Asynchronous or synchronous. For example we are creating Asynchronously.
Train the Dataset
Next we will train our dataset to identify the images. We will get Dataset ID from previous request.
HttpRequest req = new HttpRequest();
req.setMethod('POST');
req.setEndpoint('https://api.einstein.ai/v2/vision/train');
req.setHeader('content-type', 'multipart/form-data; charset="UTF-8"; boundary="1ff13444ed8140c7a32fc4e6451aa76d"');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setHeader('Cache-Control', 'no-cache');
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('name','Car vs Bike Model');
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('datasetId', <DATASET ID>);
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
req.setBodyAsBlob(formBlob);
req.setHeader('Connection', 'keep-alive');
req.setHeader('Content-Length', contentLength);
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);This command train the Dataset and create a Model. Model creation process takes time based on number of images which we have provided. In our example number of picture is less so it will complete early.
So we will make a request to check the status.
HttpRequest req = new HttpRequest();
req.setMethod('GET');
req.setEndpoint('https://api.einstein.ai/v2/vision/train/<MODEL ID>');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);Make Prediction
Once the Dataset is trained we are ready to make our first prediction. So for prediction we will use below image
HttpRequest req = new HttpRequest();
req.setMethod('POST');
req.setEndpoint('https://api.einstein.ai/v2/vision/predict');
req.setHeader('content-type', 'multipart/form-data; charset="UTF-8"; boundary="1ff13444ed8140c7a32fc4e6451aa76d"');
req.setHeader('Authorization', 'Bearer <TOKEN>');
req.setHeader('Cache-Control', 'no-cache');
// Compose the form
string form64 = '';
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('sampleLocation','https://newstechnologystuff.com/car.jpg');
form64 += HttpFormBuilder.WriteBoundary();
form64 += HttpFormBuilder.WriteBodyParameter('modelId', '<MODEL ID>');
form64 += HttpFormBuilder.WriteBoundary(HttpFormBuilder.EndingType.CrLf);
blob formBlob = EncodingUtil.base64Decode(form64);
string contentLength = string.valueOf(formBlob.size());
req.setBodyAsBlob(formBlob);
req.setHeader('Content-Length', contentLength);
req.setTimeout(60*1000);
Http h = new Http();
String resp;
HttpResponse res = h.send(req);
resp = res.getBody();
system.debug(resp);And the response we get is
So we get Almost 100% for this image. Similarly you can make your own Dataset and can play with them.
Let me know what you like most about Salesforce Einstein Vision in comments. In addition, if you want to add something share with me in comments section.
Check how we can use Einstein OCR here.
Happy Programming 🙂
4 thoughts on “Salesforce Einstein Vision: Quick Overview”