{"id":261546,"date":"2024-10-03T01:07:29","date_gmt":"2024-10-03T01:07:29","guid":{"rendered":"https:\/\/michigandigitalnews.com\/index.php\/2024\/10\/03\/pytorch-machine-learning-models-on-android\/"},"modified":"2025-06-25T17:11:00","modified_gmt":"2025-06-25T17:11:00","slug":"pytorch-machine-learning-models-on-android","status":"publish","type":"post","link":"https:\/\/michigandigitalnews.com\/index.php\/2024\/10\/03\/pytorch-machine-learning-models-on-android\/","title":{"rendered":"PyTorch machine learning models on Android"},"content":{"rendered":"

[ad_1]
\n<\/p>\n

\n
\n<\/p>\n

Posted by Paul Ruiz \u2013 Senior Developer Relations Engineer<\/em><\/p>\n

<\/a><\/p>\n

Earlier this year we launched Google AI Edge<\/a>, a suite of tools with easy access to ready-to-use ML tasks, frameworks that enable you to build ML pipelines, and run popular LLMs and custom models \u2013 all on-device. For AI on Android Spotlight Week, the Google team is highlighting various ways that Android developers can use machine learning to help improve their applications.<\/p>\n

In this post, we’ll dive into Google AI Edge Torch, which enables you to convert PyTorch models to run locally on Android and other platforms, using the Google AI Edge LiteRT<\/a> (formerly TensorFlow Lite) and MediaPipe Tasks<\/a> libraries. For insights on other powerful tools, be sure to explore the rest of the AI on Android Spotlight Week<\/a> content.<\/p>\n

To get started with Google AI Edge easier, we’ve provided samples<\/a> available on GitHub as an executable codelab. They demonstrate how to convert the MobileViT<\/a> model for image classification (compatible with MediaPipe Tasks) and the DIS<\/a> model for segmentation (compatible with LiteRT).<\/p>\n

<\/p>\n

\"a<\/div>\n

DIS model output<\/em><\/center><\/imgcaption><\/image><\/p>\n

This blog guides you through how to use the MobileViT model with MediaPipe Tasks. Keep in mind that the LiteRT runtime provides similar capabilities, enabling you to build custom pipelines and features.<\/p>\n

Convert MobileViT model for image classification compatible with MediaPipe Tasks<\/h3>\n

Once you’ve installed the necessary dependencies and utilities for your app, the first step is to retrieve the PyTorch model you wish to convert, along with any other MobileViT components you might need (such as an image processor for testing).<\/p>\n

\n
from transformers import MobileViTImageProcessor, MobileViTForImageClassification\n\nhf_model_path=\"apple\/mobilevit-small\"\nprocessor = MobileViTImageProcessor.from_pretrained(hf_model_path)\npt_model = MobileViTForImageClassification.from_pretrained(hf_model_path)\n<\/pre>\n<\/div>\n
Since the end result of this tutorial should work with MediaPipe Tasks, take an extra step to match the expected input and output shapes for image classification to what is used by the MediaPipe image classification Task.<\/p>\n
\n
class<\/span> HF2MP_ImageClassificationModelWrapper<\/span>(nn.Module):\n\n  def __init__<\/span>(self, hf_image_classification_model, hf_processor):\n    super().__init__()\n    self.model = hf_image_classification_model\n    if<\/span> hf_processor.do_rescale:\n      self.rescale_factor = hf_processor.rescale_factor\n    else<\/span>:\n      self.rescale_factor = 1.0<\/span>\n\n  def forward<\/span>(self, image: torch.Tensor):\n    # BHWC -> BCHW.\n    image = image.permute(0<\/span>, 3<\/span>, 1<\/span>, 2<\/span>)\n    # RGB -> BGR.\n    image = image.flip(dims=(1<\/span>,))\n    # Scale [0<\/span>, 255<\/span>] -> [0<\/span>, 1<\/span>].\n    image = image * self.rescale_factor\n    logits = self.model(pixel_values=image).logits  # [B, 1000<\/span>] float32.\n    # Softmax is<\/span> required for<\/span> MediaPipe classification model.\n    logits = torch.nn.functional.softmax(logits, dim=-1<\/span>)\n\n    return<\/span> logits\n<\/pre>\n<\/div>\n
<\/p>\n
\n
hf_model_path=\"apple\/mobilevit-small\"\nhf_mobile_vit_processor = MobileViTImageProcessor.from_pretrained(hf_model_path)\nhf_mobile_vit_model = MobileViTForImageClassification.from_pretrained(hf_model_path)\nwrapped_pt_model = HF2MP_ImageClassificationModelWrapper(\nhf_mobile_vit_model, hf_mobile_vit_processor).eval()\n<\/pre>\n<\/div>\n
Whether you plan to use the converted MobileViT model with MediaPipe Tasks or LiteRT, the next step is to convert the model to the .tflite<\/span> format.<\/p>\n
First, match the input shape. In this example, the input shape is 1, 256, 256, 3<\/i> for a 256×256 pixel three-channel RGB image.<\/p>\n
Then, call AI Edge Torch’s convert<\/span> function to complete the conversion process.<\/p>\n
\n
import ai_edge_torch\n\nsample_args = (torch.rand((1<\/span>, 256<\/span>, 256<\/span>, 3<\/span>)),)\nedge_model = ai_edge_torch.convert(wrapped_pt_model, sample_args)\n<\/pre>\n<\/div>\n
After converting the model, you can further refine it by incorporating metadata for the image classification labels. MediaPipe Tasks will utilize this metadata to display or return pertinent information after classification.<\/p>\n
\n
from mediapipe.tasks.python.metadata.metadata_writers import image_classifier\nfrom mediapipe.tasks.python.metadata.metadata_writers import metadata_writer\nfrom mediapipe.tasks.python.vision.image_classifier import ImageClassifier\nfrom pathlib import Path\n\nflatbuffer_file = Path('hf_mobile_vit_mp_image_classification_raw.tflite')\nedge_model.export(flatbuffer_file)\ntflite_model_buffer = flatbuffer_file.read_bytes()\n\n\/\/Extract the image classification labels from the HF models for later integration into the TFLite model.<\/span>\nlabels = list(hf_mobile_vit_model.config.id2label.values())\n\nwriter = image_classifier.MetadataWriter.create(\n    tflite_model_buffer,\n    input_norm_mean=[0.0<\/span>], #  Normalization is<\/span> not needed for<\/span> this<\/span> model.\n    input_norm_std=[1.0<\/span>],\n    labels=metadata_writer.Labels().add(labels),\n)\ntflite_model_buffer, _ = writer.populate()\n<\/pre>\n<\/div>\n
With all of that completed, it’s time to integrate your model into an Android app. If you’re following the official Colab notebook, this involves saving the model locally. For an example of image classification with MediaPipe Tasks, explore the GitHub<\/a> repository. You can find more information in the official Google AI Edge documentation<\/a>.<\/p>\n
<\/p>\n
\"moving<\/div>\n
Newly converted ViT model with MediaPipe Tasks<\/em><\/center><\/imgcaption><\/image><\/p>\n
After understanding how to convert a simple image classification model, you can use the same techniques to adapt various PyTorch models for Google AI Edge LiteRT or MediaPipe Tasks tooling on Android.<\/p>\n
For further model optimization, consider methods like quantizing<\/a> during conversion. Check out the GitHub example<\/a> to learn more about how to convert a PyTorch image segmentation model to LiteRT and quantize it.<\/p>\n
What’s Next<\/h3>\n
To keep up to date on Google AI Edge developments, look for announcements on the Google for Developers YouTube channel<\/a> and blog<\/a>.<\/p>\n
We look forward to hearing about how you’re using these features in your projects. Use #AndroidAI hashtag to share your feedback or what you’ve built in social media and check out other content in AI on Android Spotlight Week<\/a>!<\/p>\n<\/div>\n
[ad_2]
\n
Source link <\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
[ad_1] Posted by Paul Ruiz \u2013 Senior Developer Relations Engineer Earlier this year we launched Google AI Edge, a suite of tools with easy access<\/p>\n","protected":false},"author":1,"featured_media":261547,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_uf_show_specific_survey":0,"_uf_disable_surveys":false,"footnotes":""},"categories":[146],"tags":[],"_links":{"self":[{"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/posts\/261546"}],"collection":[{"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/comments?post=261546"}],"version-history":[{"count":0,"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/posts\/261546\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/media\/261547"}],"wp:attachment":[{"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/media?parent=261546"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/categories?post=261546"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/michigandigitalnews.com\/index.php\/wp-json\/wp\/v2\/tags?post=261546"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}