Building a Free Code Assistant

Prince Addo, Verrazzano Class of 2024, completed major in Computer Science and minor in Mathematics

For my capstone I worked on a free and open-source AI(Artificial Intelligence) code assistant. More specifically, I built a free and open-source neovim plugin called ollama-pilot.nvim that uses the ollama framework for open-source model integration. Although AI code assistant’s tools have quickly become popular amongst programmers, their close-source nature has been contentious. My project is novel in that it is the first implementation of a completely open-source code assistant.

I’ve had a real interest in hacking ever since taking computer science seriously after graduating high school. The idea of creating clever tools to break into systems piqued my interest; it was a mixture of technology and art, which I was exploring at the time. In order to successfully break into a system, like a true hacker, you must first understand the ins-and-outs of a system; only then can you creatively break it—like Picasso once said, “learn the rules like a pro, so you can break them like an artist.” Although I was never able to fully pursue hacking, this interest led to me learning about software development, Linux, and most importantly, open-source.

To me, open-source is a positive and peaceful place; there exists bad actors and people with ulterior motives, but similar to capitalism or blockchain algorithms, unless the majority of the system consists of bad actors, the system will function as intended. In computer systems, the source-code is the truth, therefore if someone hides or obfuscates the source-code then they might be lying—and one can argue that they probably are. As the name implies, open-source is about making the source-code open to the public to view, therefore it is the ultimate expression of honesty that can be made. If the source-code is available, and you want to know whether a program is not doing anything malicious, you can just read the code. If it’s not available, you have to operate on the trust-me-bro principle, which is infamous for being precarious in the computer industry.

Open-source is also a place to share and collaborate on projects with like-minded people. This builds confidence and a sense of community, which is important since the practice of programming is often isolating. I’ve met some of my closest friends working on open-source projects. Open-source can be taken further by making it free; the term free is not free as in price, all open-source projects are by nature free, but as in the freedom for anybody to modify the source-code in whichever way they want. If any security concern was introduced, instead of being only able to detect it, they would be able to remove it.

I came across Vim, a free and open-source project, really early in my programming journey. Once you’re in the realm of Linux, Vim is bound to come across your radar. It took me a while to appreciate it, but as I got more proficient, I gained more respect for it. It always felt a little too difficult, so I was never able to fully dive into it until NeoVim came out. NeoVim takes Vim and adds more features to, features which for one, make it more accessible to new programmers. On top of that, package managers like lazy.nvim that recently came out have made the experience of plugin management a lot easier and intuitive.

After the release of ChatGPT and the subsequent release of all things AI related, I started exploring AI and AI tools and began using my first code assistant called Codeium. I loved using it because it simplified the development process, but I couldn’t really trust it. I knew that my data was being sent to the Codeium servers and the code for those servers are not open-source, therefore who knows what they are doing with it. I had to shut it off and I eventually just stopped using it. I looked for some solution, but there was none, so I endeavored to build one, which is where this project comes in. Open-source AI is at its infancy and has several problems. One is that it requires an expensive computer to run it in a reasonable amount of time—or sometimes run at all. Another problem is that it is difficult to host. The latter problem was solved once I found the ollama framework, which is essentially a framework that allows you to host open-source models. The former problem is much harder, but will probably be fixed in the coming years. Once the major roadblocks were gone all that was left to do was architect and write the code. This proved to be very difficult as the learning curve was steep, but I committed myself to breaking the project into components and tackling each component in the development process. As of this writing, I have not completed the project, but I plan on getting at least an alpha or beta version out there.

The Health of AI in the Medical Field

Robert Wetz, Verrazzano Class of 2024, completed major in Computer Science

My thesis concerned the use of Artificial Intelligence (AI) in the medical field. I decided to choose writing a thesis for my capstone as it is something I had never done before, and I believed that I would be able to cover my topic in depth.

Upon starting my thesis, I was completely lost on where to begin. I knew what I wanted to write about and had done some research already, but starting it had proven to be the most difficult part. It was not until I met with my mentor that I was steered in the right direction and had a path to follow.

With that guidance, I began my thesis and made some headway. However, having never written a thesis before, I was unaware of all the different aspects there were in comparison to that of a college paper, something that I was very familiar with at that point in my college career. The most notable difference was how references were handled and how much they were emphasized in a thesis. Although it slowed my progress down a bit, it added a level of professionality that I had never considered outside of the references that I used to in college papers.

During this process, I also learned about the resources available through the CSI database and how expansive it is, encompassing a large number of topics. The final and most important part that I discovered when working on my capstone was simply the dedication it takes to complete. When starting out, I felt overwhelmed by the amount of content I would have to write about and I had no idea if I would even be able to complete something of this size. However, along with my mentor’s assistance, I was able to keep at it and took it piece by piece and soon enough the end was in sight. 

Something I was unprepared for in this process was how rewarding it would feel to complete it. Being able to go from nothing at all to something tangible and be a part of that process left me feeling accomplished in what I have done. Although my thesis is completed, there is room for future developments.

My thesis discusses various aspects of AI in healthcare, including its work in the field of cancer, radiology, and drug discovery. However, that only scratches the surface that AI has made within such a vast field. If I were to continue this research, different applications of AI in the medical field could be discussed or even new innovations that have come out since completing my thesis. 

Overall, I am grateful to have completed this capstone because I learned not only about AI’s impact in the field of medicine, but I also learned a lot about myself in the process.

Open AI: Elevating Website Interaction Through Chatbot

Unaiza Nizami, Verrazzano Class of 2024, completed major in Computer Science and minor in Mathematics

Research papers take a lot of time and effort, and one hopes it will be a great learning experience. When deciding on my research paper topic as a computer science major, I initially wanted to begin my capstone at the start of my senior year, but didn’t begin it until my final semester.

My research topic originated from my senior seminar class I that took on ChatGPT. As you are probably aware, ChatGPT is a popular website that many people enjoy using. In my seminar class, I concentrated on the impact of ChatGPT, the architecture it is built on, and how it came to be. For my project, I wanted to take this topic a step further. Despite the fact that the website has sparked a lot of security and educational concerns, I wanted to use it to make websites more accessible.

Many people believe that this tool is not useful because it provides most of the answers, whether the question is about history or how to make a website. This tool can solve the problem and provide the answer, but using the tool to benefit you is a skill that only a few have mastered.

This capstone project focused on the enhancement of a website and how to make it more user-friendly. I examined the CSI website to see how it could be improved, and I found that the only thing missing was a bot that could direct students to the proper location or provide them with the necessary information. Our college website has a lot of pages and information, and students frequently have trouble navigating it.

Another part of this project was researching how to do it. Since ChatGPT has been out for a little while and still improving, there were not a lot of resources on how to approach this problem. I found some YouTube videos and the open AI documentation aspect of this project to help my research. The most enjoyable aspect of my project was coding the server-side code, which was implemented as a chatbot. With the resources provided, I found it to be quite simple to follow along.

I wish I had started my project earlier or had more time to complete it. If I were to give some general advice to students undertaking a research project, it would be to start planning early. If you leave it to the last semester of your college career, you will be under a lot of pressure to complete everything in a rush. It is better to start early and give yourself plenty of time to go through the research process, find the right resources, and learn thoroughly.

Research projects are an opportunity to learn new material and challenge yourself. They can be a highlight of your college experience, and you might learn something you never thought possible.

Creating an AI Powered Color Matching Wardrobe

Steven Ceron, Verrazzano Class of 2024, completed major in Engineering Science

As I reflect on my senior engineering design project, which centered on creating an innovative clothing matching and coordination solution, I am imbued with a sense of achievement and enriched learning. The goal of my project was to design an AI-powered Color Matching Wardrobe, addressing the common frustration of not knowing what to wear in the morning.

The culmination of my four years of undergraduate education enabled me to apply the technical knowledge I had acquired. This project, which integrated my understanding of electrical engineering and computer science, was rewarding as I saw the skills and knowledge I had gained materialize in a functioning prototype. In terms of electrical engineering, ensuring that the circuit operated without mishaps was crucial. Potential risks included short circuits or component damage, although these were unlikely due to the low voltage used. To prevent such issues, it was imperative to select the correct components, requiring calculations with specific formulas, and to adhere to strict engineering safety measures. On the computer science front, the ability to create and modify code to function precisely as desired was key. Employing machine learning was essential to my project since it formed the backbone of the design. The AI needed to accurately detect objects, which, in my case, were clothes and their colors. This was much easier said than done.

The most challenging aspect of this project was undoubtedly the software component, more so than the hardware. Throughout the project, I encountered numerous problems that halted my progress for days until I found a solution. Tasks I initially considered straightforward, like installing certain programs, turned out to be the most troublesome and time-consuming. Moreover, there were issues with the coding itself. Minor changes made to improve user-friendliness often caused the program to malfunction. For instance, increasing the resolution of the live feed resulted in label boxes misaligning with the objects.

This experience has taught me that, even as I near the end of my undergraduate education, there is still much for me to learn and improve upon. However, these struggles have been enlightening. I have learned various methods to solve these issues and the importance of broadening my thought process. Often, I was overly fixated on a single solution, overlooking simpler fixes that didn't initially occur to me. The project also taught me the value of patience; rushing through certain tasks, which needed slow and precise development, led to starting over from scratch after a rushed error, a completely demoralizing experience. Yet, this has made me more cautious and attentive to ensure proper execution, helping me avoid similar setbacks in the future.

I can confidently say that, despite its stressful nature, this project has been the most enjoyable and rewarding experience of my student life. I am grateful for all the support I received, especially from my mentor Professor Changmin Kim, whose pressure was instrumental in ensuring the project's timely completion.

AI and Machine Learning with Robotics

 Avi Szczupakiewicz, Class of 2023, completed major in Engineering Science

The journey began with brainstorming for everyday problems and common issues. After reaching out to a former teacher and mentor, I was given the answer which was about people in their school putting recyclables in the wrong bins. I decided that Artificial Intelligence and Machine Learning with Robotics was the answer.

Through my research, I found a program that allowed me to capture videos of the items I wanted to be trained and recognized. Unfortunately, the program lost support, and I couldn't export the file, so I had to scrap that idea. I then found a Google Colab file that allowed me to create an Object Detection Model, but it was a slow and tedious process that had to be done manually.

After hours of manually training the model, I had the files I needed. I then reused many parts the school already had, including a Raspberry Pi 3, rails, and a motor, which helped me save money. However, I had to purchase a few bins, a second motor, and a stepper driver, and I already owned the rest of the necessary parts like wires and plugs.

I believe that future adaptations of this project could benefit from a Raspberry Pi 4 as AI is computationally demanding. At the current state of the project, I have the proof of concept working on a small scale and in the future it could be adapted to full sized bins and in every school and office.

Overall, I learned that senior projects are an essential milestone in the academic life of students, culminating all the knowledge and skills they have acquired throughout their academic journey. Through the development of the Smart Recycle project, I gained knowledge of Artificial Intelligence, Machine Learning, and hardware components including cameras, sensors, motors, and microcontrollers, as well as the development of custom software to control and coordinate these components.

I am confident that the Smart Recycle project has the potential to improve the efficiency and accuracy of recycling, reducing contamination and promoting sustainability, and I am proud of the work I have accomplished throughout its development.