AI-based auto-tagging

You can add AI-generated tags (AITags) to your media library images using ImageKit's update and upload APIs. ImageKit leverages powerful label detection APIs by Google Cloud Vision and Amazon Rekognition provided through its extensions to automatically detect and add tags to your images.
How to use auto-tagging?
1.While uploading a new image:
During an upload API call, you must specify in the extensions parameter the name of the auto-tagging extension(s) to be used along with its minConfidence and maxTags parameters.\
2.While updating an already existing image:
You can add AITags to an existing image through an update API call. You must specify in the extensions parameter the name of the auto-tagging extension(s) to be used along with its minConfidence and maxTags parameters.
Extensions parameter for auto-tagging
The extensions parameter of both upload and update APIs takes an array of objects specifying the extensions to be used along with their parameters. Each object in this array is a self-contained block containing information about one single extension. This information includes the name of the extension and other parameters associated with that extension. The object for an auto-tagging extension will have the following fields:
Parameter name
Type
Required
Description
name
string
Yes
google-auto-tagging or aws-auto-tagging
minConfidence
number
Yes
Tags with a confidence value below this value will be discarded. Valid values are between 0 and 100.
maxTags
number
Yes
Maximum number of tags to attach. If maxTags is 5, then only the top five tags returned from the API will be attached. Valid values are between 0 and 30.
Here's an example of an extensions parameter formation that will result in the execution of two different extensions - auto-tagging using Google Cloud Vision and Amazon Rekognition:
1
// Request body for an update/upload API call
2
{
3
    /*
4
    ...rest of the update/upload request parameters
5
    */
6
    "extensions": [
7
        {
8
            "name": "aws-auto-tagging",
9
            "minConfidence": 80, // only tags with a confidence value higher than 80% will be attached
10
            "maxTags": 10 // a maximum of 10 tags from aws will be attached
11
        },
12
        {
13
            "name": "google-auto-tagging",
14
            "minConfidence": 70, // only tags with a confidence value higher than 70% will be attached
15
            "maxTags": 10 // a maximum of 10 tags from google will be attached
16
        }
17
    ]
18
}
Copied!
Asynchronous behavior
Auto-tagging extensions may execute asynchronously, i.e., after your main update/upload call has successfully completed and you have received the response for it. However, auto-tagging extensions are first tried to be performed in sync. If performed synchronously, then the response of your upload/update API call will include the AITags field containing the newly generated tags. However, there is no guarantee that any extension will be performed synchronously. You can inspect the response field extensionStatus to identify the status of each extension in your request.
Response structure
If an update/upload API call includes a valid extensions field array with a non-zero length, then the response will include an extensionStatus field object, which will contain the status of each extension at the time of completion of the update/upload request. For auto-tagging extensions, the status has three possible values:
success: The extension has been successfully applied.
failed: The extension has failed and will not be retried.
pending: The extension will finish processing in some time. On completion, the final status (success/failure) will be sent to the webhookUrl provided.
The response for an API call with the above request body can look like this
1
// Response body for an update/upload API call
2
{
3
    /*
4
    ...rest of the update/upload response fields
5
    */
6
    "AITags": [
7
        {
8
            "name": "Shirt",
9
            "confidence": 90.12,
10
            "source": "google-auto-tagging"
11
        },
12
        /* ... more googleVision tags ... */
13
    ],
14
    "extensionStatus": {
15
        "google-auto-tagging": "success",
16
        "aws-auto-tagging": "pending"
17
    }
18
}
Copied!
The AITags field is populated only because the google-auto-tagging extension executed synchronously i.e. it received a successresponse. The aws-auto-taggingextension has not yet completed execution and hence its tags are not included in this response.
Webhooks
You can include a webhookUrl parameter in your update/upload API calls. The final status of extensions after they have completed execution will be delivered to this endpoint as a POST request. The request body sent to this endpoint will look like the examples below.
To learn more about how webhooks works, refer this.
1
// A SUCCESS webhook request for a google auto-tagging extension
2
{
3
    "service": "google-auto-tagging",
4
    "status": "SUCCESS",
5
    "fileObject": {
6
        "fileId": "5effaa5662679b5af2c58829",
7
        "AITags": [
8
            {
9
                "name": "Shirt",
10
                "confidence": 90.12,
11
                "source": "google-auto-tagging"
12
            },
13
            /* ... more google-auto-tagging tags ... */
14
        ],    
15
        /*
16
        ... rest of the fields in a file object
17
        */
18
    },
19
}
Copied!
1
// A FAILURE webhook request for an aws auto-tagging extension
2
{
3
    "service": "aws-auto-tagging",
4
    "status": "FAILURE",
5
    "fileId": "5effaa5662679b5af2c58829",
6
    "error": {
7
        /* error description fields */
8
    }
9
}
Copied!
Examples
Let's apply the google-auto-tagging extension to the below image:
Using upload API to add AITags
cURL
1
curl -X POST "https://upload.imagekit.io/api/v1/files/upload" \
2
-u your_private_api_key: \
3
-F '[email protected]/Users/username/Desktop/cafe_picture.jpg;type=image/jpg' \
4
-F 'fileName=my_file_name.jpg'
5
-F 'extensions="[
6
  {
7
      \"name\": \"google-auto-tagging\",
8
      \"minConfidence\": 50,
9
      \"maxTags\": 5
10
  },
11
  {
12
      \"name\": \"aws-auto-tagging\",
13
      \"minConfidence\": 50,
14
      \"maxTags\": 5
15
  },
16
]"'
Copied!
Response
1
{
2
    "fileId" : "598821f949c0a938d57563bd",
3
    "name": "file1.jpg",
4
    "url": "https://ik.imagekit.io/your_imagekit_id/images/products/file1.jpg",
5
    "thumbnailUrl": "https://ik.imagekit.io/your_imagekit_id/tr:n-media_library_thumbnail/images/products/file1.jpg",
6
    "height" : 300,
7
    "width" : 200",
8
    "size" : 83622,
9
    "filePath": "/images/products/file1.jpg",
10
    "AITags": [
11
        {
12
            "name": "Teddy Bear",
13
            "confidence": 98.52,
14
            "source": "aws-auto-tagging"
15
        },
16
        {
17
            "name": "Toy",
18
            "confidence": 98.52,
19
            "source": "aws-auto-tagging"
20
        },
21
        {
22
            "name": "Sweets",
23
            "confidence": 77.84,
24
            "source": "aws-auto-tagging"
25
        },
26
        {
27
            "name": "Heart",
28
            "confidence": 61.19,
29
            "source": "aws-auto-tagging"
30
        },
31
        {
32
            "name": "Plush",
33
            "confidence": 56.64,
34
            "source": "aws-auto-tagging"
35
        },
36
        {
37
            "name": "Toy",
38
            "confidence": 91.73,
39
            "source": "google-auto-tagging"
40
        },
41
        {
42
            "name": "Teddy bear",
43
            "confidence": 83.52,
44
            "source": "google-auto-tagging"
45
        },
46
        {
47
            "name": "Red",
48
            "confidence": 81.77,
49
            "source": "google-auto-tagging"
50
        },
51
        {
52
            "name": "Stuffed toy",
53
            "confidence": 76.25,
54
            "source": "google-auto-tagging"
55
        },
56
        {
57
            "name": "Art",
58
            "confidence": 72.83,
59
            "source": "google-auto-tagging"
60
        }
61
    ],
62
    "fileType": "image",
63
    "extensionStatus": {
64
        "google-auto-tagging": "success",
65
        "aws-auto-tagging": "success"
66
​
67
    }
68
}
Copied!
Using update API to add AITags
cURL
Node.js
Python
PHP
Ruby
1
# The unique fileId of the uploaded file. fileId is returned in response of list files API and upload API.
2
curl -X PATCH "https://api.imagekit.io/v1/files/:fileId/details" \
3
-H 'Content-Type: application/json' \
4
-u your_private_key: -d'
5
{
6
    "extensions": [ 
7
         { 
8
             "name": "google-auto-tagging",
9
             "minConfidence": 50, 
10
             "maxTags": 5
11
         },
12
          { 
13
             "name": "aws-auto-tagging",
14
             "minConfidence": 50, 
15
             "maxTags": 5
16
         }
17
     ]
18
}
19
'
Copied!
1
var ImageKit = require("imagekit");
2
​
3
var imagekit = new ImageKit({
4
    publicKey : "your_public_api_key",
5
    privateKey : "your_private_api_key",
6
    urlEndpoint : "https://ik.imagekit.io/your_imagekit_id/"
7
});
8
​
9
imagekit.updateFileDetails("file_id", { 
10
    extensions: [
11
        {
12
            name: "google-auto-tagging",
13
            maxTags: 5,
14
            minConfidence: 50
15
        },
16
        {
17
            name: "aws-auto-tagging",
18
            maxTags: 5,
19
            minConfidence: 50
20
        }
21
    ]
22
}, function(error, result) { 
23
    if(error) console.log(error);
24
    else console.log(result);
25
});
Copied!
1
from imagekitio import ImageKit
2
​
3
imagekit = ImageKit(
4
    public_key='your_public_api_key',
5
    private_key='your_private_api_key',
6
    url_endpoint = 'https://ik.imagekit.io/your_imagekit_id/'
7
)
8
​
9
updated_detail = imagekit.update_file_details(
10
    "file_id",
11
    "extensions": [
12
        {
13
            "name": "google-auto-tagging",
14
            "maxTags": 5,
15
            "minConfidence": 50
16
        },
17
        {
18
            "name": "aws-auto-tagging",
19
            "maxTags": 5,
20
            "minConfidence": 50
21
        }
22
    ]
23
)
24
​
25
print("Updated detail-", updated_detail, end="\n\n")
Copied!
1
use ImageKit\ImageKit;
2
​
3
$public_key = "your_public_api_key";
4
$your_private_key = "your_private_api_key";
5
$url_end_point = "https://ik.imagekit.io/your_imagekit_id";
6
​
7
$imageKit = new ImageKit(
8
    $public_key,
9
    $your_private_key,
10
    $url_end_point
11
);
12
​
13
$updateFileDetails = $imageKit->updateFileDetails("file_id", array("extensions" => [array("name" => "google-auto-tagging", "maxTags" => 5, "minConfidence" => 50), array("name" => "aws-auto-tagging", "maxTags" => 5, "minConfidence" => 50)]));
14
​
15
echo("Updated detail : " . json_encode($updateFileDetails));
Copied!
1
imagekitio = ImageKitIo::Client.new("your_private_key", "your_public_key", "your_url_endpoint")
2
updated_detail = imagekitio.update_file_details(
3
    file_id: "file_id_xyz",
4
    extensions: [
5
       {
6
          "name": "google-auto-tagging",
7
          "maxTags": 5,
8
          "minConfidence": 50
9
       },
10
       {
11
          "name": "aws-auto-tagging",
12
          "maxTags": 5,
13
          "minConfidence": 50
14
       }
15
    ]
16
)
Copied!
Response
1
{
2
    "fileId": "598821f949c0a938d57563bd",
3
    "type": "file",
4
    "name": "file1.jpg",
5
    "filePath": "/images/products/file1.jpg",
6
    "tags": null,
7
    "AITags": [
8
        {
9
            "name": "Teddy Bear",
10
            "confidence": 98.52,
11
            "source": "aws-auto-tagging"
12
        },
13
        {
14
            "name": "Toy",
15
            "confidence": 98.52,
16
            "source": "aws-auto-tagging"
17
        },
18
        {
19
            "name": "Sweets",
20
            "confidence": 77.84,
21
            "source": "aws-auto-tagging"
22
        },
23
        {
24
            "name": "Heart",
25
            "confidence": 61.19,
26
            "source": "aws-auto-tagging"
27
        },
28
        {
29
            "name": "Plush",
30
            "confidence": 56.64,
31
            "source": "aws-auto-tagging"
32
        },
33
        {
34
            "name": "Toy",
35
            "confidence": 91.73,
36
            "source": "google-auto-tagging"
37
        },
38
        {
39
            "name": "Teddy bear",
40
            "confidence": 83.52,
41
            "source": "google-auto-tagging"
42
        },
43
        {
44
            "name": "Red",
45
            "confidence": 81.77,
46
            "source": "google-auto-tagging"
47
        },
48
        {
49
            "name": "Stuffed toy",
50
            "confidence": 76.25,
51
            "source": "google-auto-tagging"
52
        },
53
        {
54
            "name": "Art",
55
            "confidence": 72.83,
56
            "source": "google-auto-tagging"
57
        }
58
    ],
59
    "isPrivateFile": false,
60
    "customCoordinates": null,
61
    "url": "https://ik.imagekit.io/your_imagekit_id/images/products/file1.jpg",
62
    "thumbnail": "https://ik.imagekit.io/your_imagekit_id/tr:n-media_library_thumbnail/images/products/file1.jpg",
63
    "fileType": "image",
64
    "mime": "image/jpeg",
65
    "width": 100,
66
    "height": 100,
67
    "size": 100,
68
    "hasAlpha": false,
69
    "createdAt": "2019-08-24T06:14:41.313Z",
70
    "updatedAt": "2019-08-24T06:14:41.313Z",
71
    "extensionStatus": {
72
        "google-auto-tagging": "success",
73
        "aws-auto-tagging": "success"
74
    }
75
}
Copied!
​
​