GroqOn: Unleash the Power of Groq’s LLMs Without an API (and without breaking the bank!)
Hey there, fellow AI enthusiasts and budget-conscious creators!
Have you ever been blown away by the capabilities of large language models (LLMs) like ChatGPT, Claude, but discouraged by the hefty subscription fees? Yeah, me too. That’s why I’m here to introduce you to GroqOn, a free and open-source project that lets you leverage Groq’s incredible speed of LPUs with open source LLMs without needing an API key.
The Inspiration Behind GroqOn
It was a sweltering summer evening, and I was hunched over my computer, wrestling with my latest project — a News app. I craved an LLM to summarize articles, but as a freelancer, subscriptions just weren’t in the cards. Gemini, the free option, felt sluggish with its one request per second limit. Frustration mounted.
Then, a lightbulb moment! I’d heard whispers about Groq’s lightning-fast LLMs, but there was no free API. So, I took on a mission: to create a way to access Groq’s speedy inference with Playwright. Fast forward few hours of tinkering, and GroqOn was born!
How GroqOn Works
GroqOn leverages Playwright to act like a human user. It opens Groq in your browser (don’t worry, you can run it headless!), logs you in (one-time setup!), selects the desired LLM model (llama3–70b: new kid in the block, llama3–8b, Gemma-8b, Mixtral:mixture of Expert 8x7b, etc. ), and fires off your queries. The responses are then captured and delivered to you in a convenient JSON format. Pretty cool, right?
The Benefits of GroqOn
- Free to Use: GroqOn is completely free tool and open-source. Just pure LLM goodness.
- Blazing Fast: Groq’s LPUs are known for their speed, and GroqOn bypasses login with adding cookies to context for an even faster experience.
- Open Source LLMs : You get to choose between latest and high performing Open source llms. e.g. Llama3–8b, Llama3–70b, Gemma-8b,
Mixtral-8x7b(Mixture of Experts)
- Privacy-Focused: GroqOn doesn’t share your data or cookies with anyone. You’re in complete control.
- Easy to Use: GroqOn offers a user-friendly CLI and Python library for seamless integration into your projects.
Getting Started with GroqOn
Getting started with GroqOn is a breeze. Just follow these simple steps:
- Install GroqOn:
pip install groqon - Make sure you have Playwright installed:
pip install playwright and playwright install firefox (for Firefox)
- Run GroqOn from the command line or integrate it into your Python scripts!
Use Groqon in you code:
from groqon.groq import groq
# pass single query
groq('how old is earth', model='llama3-70b')
# pass list of query
groq(["Is aunt may peter parker's actual mother?", "kya gangadhar hi shaktimaan hai?"], model='llama3-70b')
# pass other parameters
groq(
'Why am I awake at 2.30 AM?',
model='llama3-70b',
cookie_file="./groq_cookie.json",
headless=False,
save_dir='./newresponses/',
save_output=True,
system_prompt="you are jarvis/vision assistant from Ironman and marvel movie, and assistant of me, call me sir",
print_output=True
)Use Groqon with cli
groq "how old is sun" "how to fuck around and find out"\
--model llama3-70b\
--cookie_file ./groq_cookie.json\
--headless\
--save_output\
--output_dir ./output/\The Future of GroqOn
GroqOn is still under development, but there’s plenty to come! We’re working on features like:
- Multi-round chat: Have a back-and-forth conversation with the LLM for more nuanced interactions.
- Improved formatting: Enhance the output with color and rich text for better readability.
- Even more features: We’re always looking for ways to make GroqOn even more powerful.
Calling All Collaborators!
GroqOn is a community project, and we welcome your contributions! If you’re a Python whiz or have a killer idea for a new feature, we’d love to have you on board.
Ready to Unleash the Power of Groq?
Head over to the GroqOn GitHub repository, Star the repository and Start using it or Contribute to it. ! And hey, if you’re feeling generous, consider buying me a coffee (or chai!) to fuel my coding endeavors.
P.S. Don’t forget to check out Groq itself — it’s an amazing platform!
#Groq #LLM #llama3 #GroqOn #inference #opensource
