February 19, 2025
By Patricia Waldron
Through her research, Jennifer Sun wants to help human experts and AI models collaborate to unearth new insights from ever-growing stockpiles of data in diverse fields.
As a graduate student, she worked with neuroscientists to develop computer vision models to decode the behaviors of mice in lab experiments – saving the researchers’ time and eye strain. Now, as a new assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science, she is working with experts, including veterinarians at the College of Veterinary Medicine. They are exploring how AI techniques, like large language models (LLMs) that can understand and generate language, and vision models that interpret photos and video, could eventually assist veterinarians in analyzing animal health records and visual data, and making diagnoses in the clinic.
While these models are still early in their development and will need careful testing and benchmarking before being deployed in the real world, Sun envisions this work may one day help us advance animal well-being, accelerate drug discovery, and even enhance animal-human communication.
How did you get interested in applying AI to improve animal welfare?
During my Ph.D., I was working with neuroscientists. Often we have an animal behavior experiment that might involve recording videos of animals like mice over dozens or hundreds of trials. Some of the recordings could be as long as days, or maybe even months, and it's going to be very difficult for a human to manually watch all those hours of video. Because storage now is relatively cheap, you can store terabytes of video data, but having the human effort to look at all those videos is really expensive. Our work as computer scientists was to help neuroscientists automatically map these videos to recognize important behaviors.
When I joined Cornell, I was introduced to researchers in the vet school who are interested in animal behavior with a goal of improving animal welfare. There are certain conditions, like footrot, and behaviors, like play, that we can detect from video data. We're interested in whether we can develop these automated systems to help veterinarians or researchers go through huge amounts of veterinary data that might be multimodal – with text as well as video, radiology, and other types of medical data – to help them make a diagnosis or to organize their huge repository of data.
Do you have any pets?
I really want a cat. My husband's family has a cat called Miki, and she's super cute. She was a rescue cat and was super scared of people at first. My husband's family brought her up, and now she's not exactly the most social cat, but she's a lot more social in that she actually lets me pet her, even though I'm kind of a stranger from her perspective. I would love to better understand Miki.
How are people interested in applying AI models to advance veterinary medicine?
One application on the language side is organizing and standardizing veterinary records. You can use LLMs to automatically categorize your records, summarize them, as well as improve the accuracy of automatic speech transcription. On the vision side, there's a lot of work in behavior analysis and medical image analysis. These models can do things like, segment out just a relevant part of the image for further analysis. Researchers are also working to detect different types of diseases from these images automatically. These applications of AI have the potential to significantly improve animal health and veterinary medicine in the years to come.
Is it more difficult to apply vision models to veterinary patients, as compared to lab mice?
In laboratory settings, the videos are easier to analyze in the sense that they're in controlled settings. The mice might even be genetically controlled – say, a consistent coat color – and the video is usually completely stationary.
When we move to the veterinary space, there are a vast range of species with diverse appearances. It also depends on where the videos are captured. If someone's capturing a video using their smartphone, that camera would have a lot of variability, both in terms of the camera properties and the resolution, as well as how the person is capturing the video. However, researchers are developing techniques to account for these variations and improve the accuracy of vision models in more diverse videos.
How might a computer vision model one day be used for diagnosis during a veterinary visit?
What would be really interesting is, if someone recorded a bunch of data of their pets – like we all have videos or images of our pets – if the model could leverage those types of data, because the veterinarian doesn't have time to watch, say, five years of data on the animal. If a video model is trained on large amounts of existing data, it could be trained to pick up on things that are important. That's one application I can think of where the model and the veterinarian could work together.
Another one is, if the veterinarian wants to find really similar cases, the model could pull records of patients with similar conditions, even at different clinics, and maybe it could help them come to a decision. And it could also potentially, in the future, be able to suggest the percent chances of other conditions co-occurring, just based on analyzing past data. The veterinarian can either use that or ignore it. Ultimately, the veterinarian should make the final decision.
Are there data privacy and security issues associated with these models?
In the case of veterinary medicine, we have to consider the privacy of the patients, for example home videos that might contain sensitive information. Because we want to preserve privacy, we would normally anonymize the data to remove identifying information. However, anonymizing data can sometimes make it harder to understand the performance of the model during development, as access to the original data is restricted. It can make model development more difficult. Ensuring data privacy and security requires a collaborative effort – veterinary clinics, software developers, and AI researchers all have a role to play in protecting sensitive data and models trained on those data.
Furthermore, there are broader questions about AI ownership and commercialization. Who actually owns the model that we're training? If all the clinics around the world put together all their data into a big, privatized database, and privacy and security was totally solved, and we trained a big model on it, how is the model commercialized, if at all? Is it open source or is access restricted? These are complex questions that require careful consideration with experts across multiple fields.
What research are you working on currently?
I'm really interested, essentially, in how to develop new AI systems that enable scientists or clinicians to have new ways of interacting with the model and accelerating discovery from data. For example, while traditional video analysis pipelines for animal behavior analysis required a large amount of training data and supervised learning, we now have vision-language models. These models can process both visual and text information, allowing them to leverage additional context to annotate behavior and represent a much larger space of behaviors in natural language. With these advancements, instead of manually annotating large amounts of data, scientists might simply describe the behavior or provide captions for a few minutes of video, and the model could then categorize the rest. Ultimately, we hope to extend these capabilities to a wide range of scientific analysis tasks, enabling scientists to more efficiently extract insights from large-scale and messy experimental data.
Also, benchmarking is still very important – so pulling together data from a lot of different sources in order to test if our model is actually truly generalizable.
I'm also helping organize the Symposium on Artificial Intelligence in Veterinary Medicine 2.0 that was started by a few collaborators in the vet school last year. I think more symposiums or conferences where people from different fields can talk is really important.
Could you use AI to study animal culture?
For human linguistics, LLMs have made huge progress in terms of how humans process language, and learning rules of human language that maybe humans might not have even known. I think that would be interesting from a culture perspective, and also for decoding different dialects between animals, for example, using a combination of LLMs, vision, and audio analysis. We already have so much video of animals recorded online. We could even try to analyze, are there rules of animal language for cats that are co-housed with another cat? How might their language evolve compared to cats who are co-housed with another human? And how does this vary around the world and over time? And what about cats who actually watch videos online? By analyzing these communication patterns, we might gain insights into the social structures and knowledge transfer within animal populations. However, decoding animal communication is a complex challenge, and we must ensure we are conducting this research ethically.
So, how long will it be until we can talk to our pets?
I feel like it probably depends on the definition of “talk.” For example, for dogs doing tricks, I feel like some people already are talking to their pets in some sense. To get to the futuristic “the cat is wearing a headset” way, my feeling is that that's more long-term. But in the middle, I think there could be a lot of really exciting intermediate stuff, like decoding behavior or vocalizations, and really exciting connections between AI, cognitive science, and neuroscience in the space of animals. I think it's definitely a possibility, especially given enough training data for a single animal. I'm excited – I want to know what Miki is thinking and when she’s hungry!
Patricia Waldron is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.