generated_files/the_melding_of_data_engineering_and_ai.md

@@ -1,35 +0,0 @@
-# Wrangling Data: A Reality Check
-
-Okay, let’s be honest. Data wrangling isn't glamorous. It’s not a sleek, automated process of magically transforming chaos into insights. It’s a messy, frustrating, and surprisingly human endeavor. Let’s break down the usual suspects – the steps we take to get even a vaguely useful dataset, and why they’re often a monumental task.
-
-**Phase 1: The Hunt**
-
-First, you’re handed a dataset. Let’s call it “Customer_Data_v2”. It’s… somewhere. Maybe a CSV file, maybe a database table, maybe a collection of spreadsheets that haven’t been updated since 2008. Finding it is half the battle. It's like searching for a decent cup of coffee in Melbourne – you know it’s out there, but it’s often hidden behind a wall of bureaucracy.
-
-**Phase 2: Deciphering the Ancient Texts**
-
-Once you *find* it, you start learning what it *means*. This is where things get… interesting. You’re trying to understand what fields represent, what units of measurement are used, and why certain columns have bizarre names (seriously, “Customer_ID_v3”?). It takes x amount of time (depends on the industry, right?). One week for a small bakery, six months for a multinational insurance company. It’s a wild ride.
-
-You’ll spend a lot of time trying to understand the business context. "CRMs" for Customer Relationship Management? Seriously? It’s a constant stream of jargon and acronyms that make your head spin.
-
-**Phase 3: The Schema Struggle**
-
-Then there’s the schema. Oh, the schema. It takes a couple of weeks to learn the schema. It’s like deciphering ancient hieroglyphics, except instead of predicting the rise and fall of empires, you’re trying to understand why a field called “Customer_ID_v3” exists. It’s a puzzle, and a frustrating one at that.
-
-**Phase 4: The Tooling Tango**
-
-You’ll wrestle with the tools. SQL interpreters, data transformation software – they’re all there, but they’re often clunky, outdated, and require a surprising amount of manual effort. It's like finding a decent cup of coffee in Melbourne – you know it’s out there, but it’s often hidden behind a wall of bureaucracy.
-
-**Phase 5: The Reporting Revelation (and Despair)**
-
-Finally, you get to the reporting tool. And cry. Seriously, who actually *likes* this part? It’s a soul-crushing exercise in formatting and filtering, and the output is usually something that nobody actually reads.
-
-**The AI Factor – A Realistic Perspective**
-
-Now, everyone’s talking about AI. And, look, I’m not saying AI is a bad thing. It’s got potential. But let’s be realistic. This will for quite some time be the point where we need people. AI can automate the process of extracting data from a spreadsheet. But it can’t understand *why* that spreadsheet was created in the first place. It can’t understand the context, the assumptions, the biases. It can’t tell you if the data is actually useful.
-
-We can use tools like datahub to capture some of this business knowledge but those tool are only as good as the people who use them. We need to make sure AI is used for those uniform parts – schema discovery, finding the tools, ugh reporting. But where the rubber hits the road… thats where we need people and that we are making sure that there is a person interpreting not only what goes out.. but what goes in.
-
-**The Bottom Line**
-
-It’s a bit like trying to build a great BBQ. You can buy the fanciest gadgets and the most expensive wood, but if you don’t know how to cook, you’re going to end up with a burnt mess. So, let’s not get carried away with the hype. Let’s focus on building a data culture that values human intelligence, critical thinking, and a good dose of common sense. And let’s keep wrangling. Because, let’s be honest, someone’s gotta do it.

generated_files/when_to_use_ai.md

@@ -1,53 +0,0 @@
-# When Should You Use AI?
-
-Right off the bat? Well, let’s talk about when *not* using an LLM is actually pretty much like trying to build that perfect pavlova with a robot: Sure, they might have all these instructions and ingredients laid out for them (or so it seems), but can you really trust this machine to understand those subtle nuances of temperature or timing? No. And let’s be real here – if we’re talking about tasks requiring precise logic like financial calculations or scientific modeling - well, that sounds more suited to the human brain.
-
-But where does AI actually shine bright and come in handy?
-
-* **Pattern Recognition:** Spotting trends within data is one of those areas LLMs are pretty darn good at. Whether it’s identifying patterns across a dataset for insights (or even generating creative ideas based on existing information), they can do that with speed, efficiency - not to mention accuracy.
-
-**And when shouldn’t you use AI?**
-
-* **Tasks Requiring Precise Logic:** If your job is something needing absolute precision – like crunching numbers or modeling scientific data where a miscalculation could mean millions in losses for the company. Well… maybe hold off on letting an LLM take over.
-* **Situations Demanding Critical Thinking**: Let’s be honest, if you need to make judgment calls based upon complex factors that even humans can struggle with – then it might not just do a good job; but rather fall short.
-
-LMLs are great at mimicking intelligence. But they don’t actually understand things the way we human beings (or I should say: non-humans) comprehend them.
-* **Processes Where Errors Have Serious Consequences:** If your work involves tasks where errors can have serious consequences, then you probably want to keep it in human hands.
-
-**The Bottom Line**
-
-AI is a powerful tool. But like any good chef knows – even the best kitchen appliances can't replace their own skills and experience when making that perfect pavlova (or for us humans: delivering results). It’s about finding balance between leveraging AI capabilities, while also relying on our critical thinking - and human intuition.
-
-Don’t get me wrong here; I’m not anti-AI. But let’s be sensible – use it where it's truly helpful but don't forget to keep those tasks in the hands of your fellow humans (or at least their non-humans).
-
----
-
-**Note for Editors:** This draft is designed with ease-of-editing and clarity as a priority, so feel free to adjust any sections that might need further refinement or expansion. I aimed this piece towards an audience who appreciates both humor-infused insights into the world of AI – while also acknowledging its limitations in certain scenarios.
-
-```markdown
-# When Should You Use AI?
-
-Right off the bat? Well, let’s talk about when *not* using LLM is actually pretty much like trying to build that perfect pavlova with a robot: Sure, they might have all these instructions and ingredients laid out for them (or so it seems), but can you really trust this machine to understand those subtle nuances of temperature or timing? No. And let’s be real here – if we’re talking about tasks requiring precise logic like financial calculations or scientific modeling - well, that sounds more suited to the human brain.
-
-But where does AI actually shine bright and come in handy?
-
-* **Pattern Recognition:** Spotting trends within data is one of those areas LLMs are pretty darn good at. Whether it’s identifying patterns across a dataset for insights (or even generating creative ideas based on existing information), they can do that with speed, efficiency - not to mention accuracy.
-
-**And when shouldn’t you use AI?**
-
-* **Tasks Requiring Precise Logic:** If your job is something needing absolute precision – like crunching numbers or modeling scientific data where a miscalculation could mean millions in losses for the company. Well… maybe hold off on letting an LLM take over.
-* **Situations Demanding Critical Thinking**: Let’s be honest, if you need to make judgment calls based upon complex factors that even humans can struggle with – then it might not just do a good job; but rather fall short.
-
-LMLs are great at mimicking intelligence. But they don’t actually understand things the way we human beings (or I should say: non-humans) comprehend them.
-* **Processes Where Errors Have Serious Consequences:** If your work involves tasks where errors can have serious consequences, then you probably want to keep it in human hands.
-
-**The Bottom Line**
-
-AI is a powerful tool. But like any good chef knows – even the best kitchen appliances can't replace their own skills and experience when making that perfect pavlova (or for us humans: delivering results). It’s about finding balance between leveraging AI capabilities, while also relying on our critical thinking - and human intuition.
-
-Don’t get me wrong here; I’m not anti-AI. But let’s be sensible – use it where it's truly helpful but don't forget to keep those tasks in the hands of your fellow humans (or at least their non-humans).
-
----
-
-**Note for Editors:** This draft is designed with ease-of-editing and clarity as a priority, so feel free to adjust any sections that might need further refinement or expansion. I aimed this piece towards an audience who appreciates both humor-infused insights into the world of AI – while also acknowledging its limitations in certain scenarios.
-```

src/ai_generators/ollama_md_generator.py

@@ -3,6 +3,7 @@ from ollama import Client
 import chromadb
 from langchain_ollama import ChatOllama
 
+
 class OllamaGenerator:
 
     def __init__(self, title: str, content: str, inner_title: str):