In this work, we delve into the domain of source code similarity detection using Large Language Models (LLMs). Our investigation is motivated by the necessity to identify similarities among different pieces of source code, a critical aspect for tasks such as plagiarism detection and code reuse. We specifically focus on exploring the effectiveness of leveraging LLMs for this purpose. To achieve this, we utilized the LLMSecEval dataset, comprising 150 NL prompts for code generation across two languages - C and Python, and employed radamsa, a mutation-based input generator, to create 27 different mutations per NL prompt. Subsequently, using Gemini Pro, we generated code for the original and mutated NL prompts to study code similarity using CodeBERT. Our experiment aims to uncover the extent to which LLMs can consistently generate similar code despite mutations in the input NL prompts, providing insights into the robustness and generalization capabilities of LLMs in understanding and comparing programming code structures and semantics.